[erlang-questions] VM leaking memory

[Fred Hebert]

On Fri, Feb 1, 2019 at 11:11 AM Frank Muller <frank.muller.erl@REDACTED>
wrote:

>
> 2. Hibernating the process after processing N packets (varying N=10..128).
> The HIT rates went above 90% immediately.
> http://147.135.36.188:3400/observer_cli_AFTER.jpg
>
> What is the effect of hibernating this process on the long term?
> This process is receiving about ~1200 packets/sec under normal load and
> can reach ~3000 packets/sec under heavy load.
>
> Is there a better way of solving the problem by tweeting the bin allocator
> SBC/MBC?
>

So hibernation will do a few things:

- a full-sweep garbage collection
- drop the stack
- memory compaction on the process.

Unless specified otherwise, a call to `erlang:garbage_collect(Pid)` forces
a major collection so it may look like what you have is more or less a case
of a process spiking with a lot of memory, and then becoming mostly idle
while still holding enough references to refc binaries to not GC old data.
Rince and repeat and you get a lot of old stuff.

Fragmentation of this kind is often resolved with settings such as `+MBas
aobf +MBlmbcs 512` being passed to the emulator, which changes the
allocation strategy for one that favors lower addresses, and reduces the
size of a multiblock carrier to use more of them. The objective of this
being to reuse existing blocks and make it easier to deallocate some by
reducing the chance some binary keeps it active.

If what you have is really one process though, you may get better results
by running some hibernation from time to time, but only experimentation
will confirm. If the allocator strategies don't cut it (or can't be used
because you want to keep the 5 years live upgrade streak going), do
something like count the packets you received, and every N of them (pick a
large value so you might run a compaction once every 10 minutes or so; you
can pick based on the leaking rate), force a hibernation to shed some
memory.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190201/d7754199/attachment.htm>