[erlang-questions] zlib driver in erlang seems very slow

[Loïc Hoguin]

Erlang/zlib using only one core fits the discussion I had about it. Not 
sure about the OOM crash though.

By the way I don't know if there's anything out there about it but Joe 
mentioned a lock in zlib here: 
https://joearms.github.io/2013/03/28/solving-the-wrong-problem.html - 
Little details, and I don't think they open sourced their zlib 
implementation though.

On 03/09/2016 01:28 PM, Caragea Silviu wrote:
> Yes so your suggestion to test concurrently was very good. I updated the
> testing script and running it on my machine with 1000 concurrent
> processes It's crashing the VM using erlang driver . I suspect because
> out of memory. My machine has only 16 GB and I see at some point 48 GB
> in the beam process.
>
> Anyway before crashing all the time I see cpu consumption being 100 %
> maximum on beam process.
>
> The same test using my library complete fine. also the cpu I see cpu
> staying at 500 % - 600 % and sometimes 700% which means it's using
> almost all the cores I have. Memory wise is not using more than : 12 GB
>
> Something is strange. You can review my testing program maybe I'm wrong
> somewhere.
>
> Silviu
>
>
>
>
> On Wed, Mar 9, 2016 at 1:15 PM, Caragea Silviu <silviu.cpp@REDACTED
> <mailto:silviu.cpp@REDACTED>> wrote:
>
>     Hello Loic,
>
>     I didn't tested concurrently but I will do it shouldn't be very
>     difficult to change the testing script. Even if from what I know
>     zlib doesn't have any mutex or something to lock the access to
>     resources inside.
>     Regarding inflateChunk I might add it, right now its only an
>     interface that makes integration in the projects I'm using faster.
>
>     Silviu
>
>     On Wed, Mar 9, 2016 at 1:08 PM, Loïc Hoguin <essen@REDACTED
>     <mailto:essen@REDACTED>> wrote:
>
>         On 03/09/2016 11:54 AM, Caragea Silviu wrote:
>
>             Some benchmark results you can see on the
>             https://github.com/silviucpp/ezlib
>
>             I can understand that the erlang driver api is generic and
>             allows you to
>             customize a lot if things and mine it's not and it's built
>             around my need to use it with ejabberd but still I'm almost
>             sure if I
>             change the interface the performance difference will be the
>             same.
>
>             Maybe my benchmark is not realistic or I'm missing something.
>             Feedback it's welcome.
>
>
>         I have recently come to realize the same thing as you. A few
>         discussions I had with users pointed out that the performance of
>         the Erlang zlib might be even worse when used concurrently. Have
>         you done some measurements on that?
>
>         Also you will probably need an inflateChunk equivalent if you
>         are doing anything public facing. I don't think it changes much
>         performance-wise though.
>
>         --
>         Loïc Hoguin
>         http://ninenines.eu
>         Author of The Erlanger Playbook,
>         A book about software development using Erlang
>
>
>

-- 
Loïc Hoguin
http://ninenines.eu
Author of The Erlanger Playbook,
A book about software development using Erlang