[erlang-questions] How can we increase multiblock carrier utilization for binary_alloc?

[Gerhard Lazu]

We are running on Erlang 20.3.4 on Linux 4.4.0-119-generic 14.04.1-Ubuntu
SMP x86_64. These are the flags that we use for beam.smp:

/var/vcap/packages/erlang-20.3.4/lib/erlang/erts-9.3/bin/beam.smp -W w -A
64 -zdbbl 128000 -K true -stbt db -zdbbl 128000 -P 1048576 -t 5000000 -MHas
ageffcbf -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512

When the node boots, the binary_alloc multiblock carrier utilization is
~99% (90MB allocated, 90MB used). [1]

As the load on the node starts, binary_alloc mbcs util drops to ~78%
(~800MB allocated, ~625MB used). [2]

When load stops completely, binary_alloc mbcs util drops to ~61%  (~55MB
allocated, ~35MB used) & mbcs_pool goes to ~36% (~215MB allocated, ~80MB
used). In the context of RabbitMQ, this is a big problem since memory usage
controls whether incoming messages are blocked or not (a.k.a. memory
alarm). It's essential that the Erlang VM utilises memory as efficiently as
possible, otherwise nodes under no load can remain blocked permanently. [3]

Our goal is for the Erlang VM to have as little unused memory as possible.
As you could see in the referenced screenshots [1][2][3], the total unused
memory starts at ~30MB and grows to ~280MB. Considering that the total RSS
memory that the beam.smp process uses is 545MB [3], half of it goes unused
(~280MB), and this is a big problem for RabbitMQ, as mentioned above.

Lukas, you've shared some excellent documentation in the past around the
topic of memory management in Erlang. I am wondering if you have
deeper/more refined insights that could help our current challenge. In
Erlang Memory Management Battle Stories [5], on slide 29, you mention
"Decreasing largest mbc size will make more carriers and hopefully be able
to free them". In our case, it doesn't, and I'm hoping that you can point
us in the right direction.

Ferd, I've studied your past battles with Erlang's memory management, and I
can only thank you for sharing so much over the years. Erlang in Anger [6]
and recon [7] helped immensely, thank you. I would appreciate greatly if
you could nudge us in the right direction, maybe we've missed something.

During this exploration, there's a specific thing that's been bugging us:
why is RSS smaller than the allocated memory?

Thank you all, Gerhard & Loïc.

[1]:
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/mbcs_utilization_after_boot.png
[2]:
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/mbcs_utilisation_under_load.png
[3]:
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/mbcs_utilisation_after_load.png
[4]: http://erlang.org/doc/man/erts_alloc.html
[5]:
http://www.erlang-factory.com/static/upload/media/139454517145429lukaslarsson.pdf
[6]: https://www.erlang-in-anger.com/
[7]: https://github.com/ferd/recon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180501/363212b5/attachment.htm>