[erlang-questions] VM leaking memory

[Frank Muller]

Hi Fred

I will implement your hibernation idea every N-minutes on monday and report
back.

Thanks again.

/Frank


>
> On Fri, Feb 1, 2019 at 11:11 AM Frank Muller <frank.muller.erl@REDACTED>
> wrote:
>
>>
>> 2. Hibernating the process after processing N packets (varying N=10..128).
>> The HIT rates went above 90% immediately.
>> http://147.135.36.188:3400/observer_cli_AFTER.jpg
>>
>> What is the effect of hibernating this process on the long term?
>> This process is receiving about ~1200 packets/sec under normal load and
>> can reach ~3000 packets/sec under heavy load.
>>
>> Is there a better way of solving the problem by tweeting the bin
>> allocator SBC/MBC?
>>
>
> So hibernation will do a few things:
>
> - a full-sweep garbage collection
> - drop the stack
> - memory compaction on the process.
>
> Unless specified otherwise, a call to `erlang:garbage_collect(Pid)` forces
> a major collection so it may look like what you have is more or less a case
> of a process spiking with a lot of memory, and then becoming mostly idle
> while still holding enough references to refc binaries to not GC old data.
> Rince and repeat and you get a lot of old stuff.
>
> Fragmentation of this kind is often resolved with settings such as `+MBas
> aobf +MBlmbcs 512` being passed to the emulator, which changes the
> allocation strategy for one that favors lower addresses, and reduces the
> size of a multiblock carrier to use more of them. The objective of this
> being to reuse existing blocks and make it easier to deallocate some by
> reducing the chance some binary keeps it active.
>
> If what you have is really one process though, you may get better results
> by running some hibernation from time to time, but only experimentation
> will confirm. If the allocator strategies don't cut it (or can't be used
> because you want to keep the 5 years live upgrade streak going), do
> something like count the packets you received, and every N of them (pick a
> large value so you might run a compaction once every 10 minutes or so; you
> can pick based on the leaking rate), force a hibernation to shed some
> memory.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190202/1fb2c665/attachment.htm>