[erlang-questions] VM leaking memory

[Frank Muller]

Hi Michael

All packets in transit have a “seq_id” (sequential number).

This means that in theory packet1, packet2...packetN can be checked in
parallel and in any order (which is not the case in my current design), but
they must be send to the next processing stage in order: packet1 first,
then packet2...

I would love to hear from you how can I turn this long-lived process to
multiple short-lived ones while enforcing ordering.

/Frank

On 2/1/19 8:11 AM, Frank Muller wrote:
>
> I tried two solutions to reduce the memory usage of the problematic
> process:
>
> 1. calling garbage:collect/0 after processing N packets (varying
> N=10..128).
> Nothing changed at all and the bin_alloc memory stayed fragmented as you
> can see:
> http://147.135.36.188:3400/observer_cli_BEFORE.jpg
>
> The call to instrument:carriers/0:
> http://147.135.36.188:3400/instrument_carriers.jpg
>
> The call to instrument:allocations/0:
> http://147.135.36.188:3400/instrument_allocations.jpg
>
>
> 2. Hibernating the process after processing N packets (varying N=10..128).
> The HIT rates went above 90% immediately.
> http://147.135.36.188:3400/observer_cli_AFTER.jpg
>
> What is the effect of hibernating this process on the long term?
> This process is receiving about ~1200 packets/sec under normal load and
> can reach ~3000 packets/sec under heavy load.
>
> Is there a better way of solving the problem by tweeting the bin allocator
> SBC/MBC?
>
>
> /Frank
>
> _______________________________________________
> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
>
> If you move the creation of temporary binaries out of any Erlang processes
> you have that are long-lived, into short-lived Erlang processes, you would
> no longer have this problem.  The tuning discussions, allocator options,
> hibernate use, etc. is not solving the cause of the problem.  Source code
> should not need to call garbage:collect/0 and using temporary Erlang
> processes makes the garbage collection occur naturally, at a pace that
> shouldn't require special tuning.
> Best Regards,
> Michael
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190202/e4812e56/attachment.htm>