[erlang-questions] zlib driver in erlang seems very slow

[Caragea Silviu]

Yes so your suggestion to test concurrently was very good. I updated the
testing script and running it on my machine with 1000 concurrent processes
It's crashing the VM using erlang driver . I suspect because out of memory.
My machine has only 16 GB and I see at some point 48 GB in the beam
process.

Anyway before crashing all the time I see cpu consumption being 100 %
maximum on beam process.

The same test using my library complete fine. also the cpu I see cpu
staying at 500 % - 600 % and sometimes 700% which means it's using almost
all the cores I have. Memory wise is not using more than : 12 GB

Something is strange. You can review my testing program maybe I'm wrong
somewhere.

Silviu




On Wed, Mar 9, 2016 at 1:15 PM, Caragea Silviu <silviu.cpp@REDACTED> wrote:

> Hello Loic,
>
> I didn't tested concurrently but I will do it shouldn't be very difficult
> to change the testing script. Even if from what I know zlib doesn't have
> any mutex or something to lock the access to resources inside.
> Regarding inflateChunk I might add it, right now its only an interface
> that makes integration in the projects I'm using faster.
>
> Silviu
>
> On Wed, Mar 9, 2016 at 1:08 PM, Loïc Hoguin <essen@REDACTED> wrote:
>
>> On 03/09/2016 11:54 AM, Caragea Silviu wrote:
>>
>>> Some benchmark results you can see on the
>>> https://github.com/silviucpp/ezlib
>>>
>>> I can understand that the erlang driver api is generic and allows you to
>>> customize a lot if things and mine it's not and it's built
>>> around my need to use it with ejabberd but still I'm almost sure if I
>>> change the interface the performance difference will be the same.
>>>
>>> Maybe my benchmark is not realistic or I'm missing something.
>>> Feedback it's welcome.
>>>
>>
>> I have recently come to realize the same thing as you. A few discussions
>> I had with users pointed out that the performance of the Erlang zlib might
>> be even worse when used concurrently. Have you done some measurements on
>> that?
>>
>> Also you will probably need an inflateChunk equivalent if you are doing
>> anything public facing. I don't think it changes much performance-wise
>> though.
>>
>> --
>> Loïc Hoguin
>> http://ninenines.eu
>> Author of The Erlanger Playbook,
>> A book about software development using Erlang
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160309/c18da560/attachment.htm>