[erlang-questions] VM leaking memory

[Frank Muller]

I tried two solutions to reduce the memory usage of the problematic process:

1. calling garbage:collect/0 after processing N packets (varying N=10..128).
Nothing changed at all and the bin_alloc memory stayed fragmented as you
can see:
http://147.135.36.188:3400/observer_cli_BEFORE.jpg

The call to instrument:carriers/0:
http://147.135.36.188:3400/instrument_carriers.jpg

The call to instrument:allocations/0:
http://147.135.36.188:3400/instrument_allocations.jpg


2. Hibernating the process after processing N packets (varying N=10..128).
The HIT rate went above 90% immediately.
http://147.135.36.188:3400/observer_cli_AFTER.jpg

What is the effect of frequent hibernating on the long term? This process
is receiving about ~1200 packets/sec under normal load and can reach ~3000
packets/sec under heavy load.

Is there a better way of solving this memory issue  by tweeting the bin
allocator SBC/MBC?

/Frank

A text file would work a lot better, the important information is missing
> from your last screenshot.
>
> This should fit binary_alloc stats for the first scheduler in a screenshot
> (your system has 48 schedulers):
>
> [_|[SCHED0|_]] = erlang:system_info({allocator, binary_alloc}),
> io:format("~p~n", [SCHED0]).
>
>
>
> On Fri, Feb 1, 2019 at 11:14 AM Frank Muller <frank.muller.erl@REDACTED>
> wrote:
>
>> Here is the beginning of recon_alloc:snapshot/1.
>> Yes, Lukas can help on this. VM’s memory management looks like a mystery.
>>
>>
>>
>> Hi Frank,
>>>
>>> The binary_alloc seems to be the one misbehaving (not 100% sure, see
>>>> attached screenshot).
>>>>
>>>
>>> I agree, it's your binary_alloc. How many single block carriers in
>>> binary_alloc do you have? What about multi-block carriers? Can you share a
>>> recon_alloc snapshot?
>>>
>>>
>>>> Is there a way to tweak this allocator strategy SBC/MBC? Lukas Larsson
>>>> maybe?
>>>>
>>>
>>> For us, RabbitMQ, defaulting to a lower multi-block carrier size (
>>> +MBlmbcs <http://erlang.org/doc/man/erts_alloc.html#M_lmbcs>) as well
>>> as changing the allocation strategy (+MBas
>>> <http://erlang.org/doc/man/erts_alloc.html#M_as>) made a positive
>>> difference. From the screenshot that you've shared, I am guessing that this
>>> won't work for you. This is the process that we went through at RabbitMQ:
>>> https://groups.google.com/forum/#!msg/rabbitmq-users/LSYaac9frYw/LNZDZUlrBAAJ
>>>
>>> I found Lukas' Erlang Memory Management Battle Stories from 2014 to be a
>>> great erts_alloc companion:
>>> http://www.erlang-factory.com/static/upload/media/139454517145429lukaslarsson.pdf
>>> & https://www.youtube.com/watch?v=nuCYL0X-8f4
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190201/1434e690/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_6342.jpg
Type: image/jpeg
Size: 199468 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190201/1434e690/attachment.jpg>