[erlang-questions] zlib driver in erlang seems very slow

[Graham Hay]

Sadly Concurix has "pivoted" to instrumenting nodejs apps.

On 9 March 2016 at 12:39, Loïc Hoguin <essen@REDACTED> wrote:
> Erlang/zlib using only one core fits the discussion I had about it. Not sure
> about the OOM crash though.
>
> By the way I don't know if there's anything out there about it but Joe
> mentioned a lock in zlib here:
> https://joearms.github.io/2013/03/28/solving-the-wrong-problem.html - Little
> details, and I don't think they open sourced their zlib implementation
> though.
>
> On 03/09/2016 01:28 PM, Caragea Silviu wrote:
>>
>> Yes so your suggestion to test concurrently was very good. I updated the
>> testing script and running it on my machine with 1000 concurrent
>> processes It's crashing the VM using erlang driver . I suspect because
>> out of memory. My machine has only 16 GB and I see at some point 48 GB
>> in the beam process.
>>
>> Anyway before crashing all the time I see cpu consumption being 100 %
>> maximum on beam process.
>>
>> The same test using my library complete fine. also the cpu I see cpu
>> staying at 500 % - 600 % and sometimes 700% which means it's using
>> almost all the cores I have. Memory wise is not using more than : 12 GB
>>
>> Something is strange. You can review my testing program maybe I'm wrong
>> somewhere.
>>
>> Silviu
>>
>>
>>
>>
>> On Wed, Mar 9, 2016 at 1:15 PM, Caragea Silviu <silviu.cpp@REDACTED
>> <mailto:silviu.cpp@REDACTED>> wrote:
>>
>>     Hello Loic,
>>
>>     I didn't tested concurrently but I will do it shouldn't be very
>>     difficult to change the testing script. Even if from what I know
>>     zlib doesn't have any mutex or something to lock the access to
>>     resources inside.
>>     Regarding inflateChunk I might add it, right now its only an
>>     interface that makes integration in the projects I'm using faster.
>>
>>     Silviu
>>
>>     On Wed, Mar 9, 2016 at 1:08 PM, Loïc Hoguin <essen@REDACTED
>>     <mailto:essen@REDACTED>> wrote:
>>
>>         On 03/09/2016 11:54 AM, Caragea Silviu wrote:
>>
>>             Some benchmark results you can see on the
>>             https://github.com/silviucpp/ezlib
>>
>>             I can understand that the erlang driver api is generic and
>>             allows you to
>>             customize a lot if things and mine it's not and it's built
>>             around my need to use it with ejabberd but still I'm almost
>>             sure if I
>>             change the interface the performance difference will be the
>>             same.
>>
>>             Maybe my benchmark is not realistic or I'm missing something.
>>             Feedback it's welcome.
>>
>>
>>         I have recently come to realize the same thing as you. A few
>>         discussions I had with users pointed out that the performance of
>>         the Erlang zlib might be even worse when used concurrently. Have
>>         you done some measurements on that?
>>
>>         Also you will probably need an inflateChunk equivalent if you
>>         are doing anything public facing. I don't think it changes much
>>         performance-wise though.
>>
>>         --
>>         Loïc Hoguin
>>         http://ninenines.eu
>>         Author of The Erlanger Playbook,
>>         A book about software development using Erlang
>>
>>
>>
>
> --
> Loïc Hoguin
> http://ninenines.eu
> Author of The Erlanger Playbook,
> A book about software development using Erlang
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions