[erlang-questions] How can we increase multiblock carrier utilization for binary_alloc?

[Lukas Larsson]

Hello,

On Tue, May 1, 2018 at 3:05 PM, Gerhard Lazu <gerhard@REDACTED> wrote:

> We are running on Erlang 20.3.4 on Linux 4.4.0-119-generic 14.04.1-Ubuntu
> SMP x86_64. These are the flags that we use for beam.smp:
>
> /var/vcap/packages/erlang-20.3.4/lib/erlang/erts-9.3/bin/beam.smp -W w -A
> 64 -zdbbl 128000 -K true -stbt db -zdbbl 128000 -P 1048576 -t 5000000 -MHas
> ageffcbf -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512
>
> When the node boots, the binary_alloc multiblock carrier utilization is
> ~99% (90MB allocated, 90MB used). [1]
>
> As the load on the node starts, binary_alloc mbcs util drops to ~78%
> (~800MB allocated, ~625MB used). [2]
>
> When load stops completely, binary_alloc mbcs util drops to ~61%  (~55MB
> allocated, ~35MB used) & mbcs_pool goes to ~36% (~215MB allocated, ~80MB
> used).
>

> In the context of RabbitMQ, this is a big problem since memory usage
> controls whether incoming messages are blocked or not (a.k.a. memory
> alarm). It's essential that the Erlang VM utilises memory as efficiently as
> possible, otherwise nodes under no load can remain blocked permanently. [3]
>
> Our goal is for the Erlang VM to have as little unused memory as possible.
> As you could see in the referenced screenshots [1][2][3], the total unused
> memory starts at ~30MB and grows to ~280MB. Considering that the total RSS
> memory that the beam.smp process uses is 545MB [3], half of it goes unused
> (~280MB), and this is a big problem for RabbitMQ, as mentioned above.
>

Why do you not use erlang:memory() as the base for whether you can accept
more messages? Having a low memory utilisation is not bad in itself, unless
of course some other program on the same machine needs the memory.

Looking at the used memory under load and after, at peak the allocated
memory is 1510 MB and then after it is 577 MB. So about 2/3rds of the
allocated memory was returned to the OS. While this is not perfect, it is
not terrible either. Reducing it further may not be easy.


>
> Lukas, you've shared some excellent documentation in the past around the
> topic of memory management in Erlang. I am wondering if you have
> deeper/more refined insights that could help our current challenge. In
> Erlang Memory Management Battle Stories [5], on slide 29, you mention
> "Decreasing largest mbc size will make more carriers and hopefully be able
> to free them". In our case, it doesn't, and I'm hoping that you can point
> us in the right direction.
>

Can you reproduce the behaviour? Would it be possible to get a recon_alloc
snaphot during and after load?

Have you seen https://github.com/erlang/otp/pull/1790 that was just merged
to master with the accompanying blog post:
http://blog.erlang.org/Memory-instrumentation-in-OTP-21/?

I assume that you have tried ageffcaobf?


>
> During this exploration, there's a specific thing that's been bugging us:
> why is RSS smaller than the allocated memory?
>

Every time I try to understand how RSS works I just end up getting more
confused.


>
> Thank you all, Gerhard & Loïc.
>
> [1]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-all
> ocators/mbcs_utilization_after_boot.png
> [2]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-all
> ocators/mbcs_utilisation_under_load.png
> [3]: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-all
> ocators/mbcs_utilisation_after_load.png
> [4]: http://erlang.org/doc/man/erts_alloc.html
> [5]: http://www.erlang-factory.com/static/upload/media/1394545171
> 45429lukaslarsson.pdf
> [6]: https://www.erlang-in-anger.com/
> [7]: https://github.com/ferd/recon
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180502/29d66227/attachment.htm>