[erlang-questions] zlib driver in erlang seems very slow
Loïc Hoguin
essen@REDACTED
Wed Mar 9 13:39:58 CET 2016
Erlang/zlib using only one core fits the discussion I had about it. Not
sure about the OOM crash though.
By the way I don't know if there's anything out there about it but Joe
mentioned a lock in zlib here:
https://joearms.github.io/2013/03/28/solving-the-wrong-problem.html -
Little details, and I don't think they open sourced their zlib
implementation though.
On 03/09/2016 01:28 PM, Caragea Silviu wrote:
> Yes so your suggestion to test concurrently was very good. I updated the
> testing script and running it on my machine with 1000 concurrent
> processes It's crashing the VM using erlang driver . I suspect because
> out of memory. My machine has only 16 GB and I see at some point 48 GB
> in the beam process.
>
> Anyway before crashing all the time I see cpu consumption being 100 %
> maximum on beam process.
>
> The same test using my library complete fine. also the cpu I see cpu
> staying at 500 % - 600 % and sometimes 700% which means it's using
> almost all the cores I have. Memory wise is not using more than : 12 GB
>
> Something is strange. You can review my testing program maybe I'm wrong
> somewhere.
>
> Silviu
>
>
>
>
> On Wed, Mar 9, 2016 at 1:15 PM, Caragea Silviu <silviu.cpp@REDACTED
> <mailto:silviu.cpp@REDACTED>> wrote:
>
> Hello Loic,
>
> I didn't tested concurrently but I will do it shouldn't be very
> difficult to change the testing script. Even if from what I know
> zlib doesn't have any mutex or something to lock the access to
> resources inside.
> Regarding inflateChunk I might add it, right now its only an
> interface that makes integration in the projects I'm using faster.
>
> Silviu
>
> On Wed, Mar 9, 2016 at 1:08 PM, Loïc Hoguin <essen@REDACTED
> <mailto:essen@REDACTED>> wrote:
>
> On 03/09/2016 11:54 AM, Caragea Silviu wrote:
>
> Some benchmark results you can see on the
> https://github.com/silviucpp/ezlib
>
> I can understand that the erlang driver api is generic and
> allows you to
> customize a lot if things and mine it's not and it's built
> around my need to use it with ejabberd but still I'm almost
> sure if I
> change the interface the performance difference will be the
> same.
>
> Maybe my benchmark is not realistic or I'm missing something.
> Feedback it's welcome.
>
>
> I have recently come to realize the same thing as you. A few
> discussions I had with users pointed out that the performance of
> the Erlang zlib might be even worse when used concurrently. Have
> you done some measurements on that?
>
> Also you will probably need an inflateChunk equivalent if you
> are doing anything public facing. I don't think it changes much
> performance-wise though.
>
> --
> Loïc Hoguin
> http://ninenines.eu
> Author of The Erlanger Playbook,
> A book about software development using Erlang
>
>
>
--
Loïc Hoguin
http://ninenines.eu
Author of The Erlanger Playbook,
A book about software development using Erlang
More information about the erlang-questions
mailing list