[erlang-questions] How can we increase multiblock carrier utilization for binary_alloc?

[Gerhard Lazu]

Hi Lukas,

Why do you not use erlang:memory() as the base for whether you can accept
> more messages? Having a low memory utilisation is not bad in itself, unless
> of course some other program on the same machine needs the memory.
>

We used to use erlang:memory(), but we've learned that it doesn't work well
in practice [1]. Linux OOM will take action based on RSS, not Erlang
allocated memory.


> Looking at the used memory under load and after, at peak the allocated
> memory is 1510 MB and then after it is 577 MB. So about 2/3rds of the
> allocated memory was returned to the OS. While this is not perfect, it is
> not terrible either. Reducing it further may not be easy.
>

Your observation is true and accurate. It's also true and accurate that out
of 577MB allocated, 300MB is used & 277MB is unused, meaning that almost
half of the allocated memory is not used.

I understand that it may not be easy to reduce the unused memory, but all
I'm thinking is that whilst this unused memory might seem small in this
particular scenario, what happens when the Erlang VM has 60GB allocated?

Would it help if we can show the impact of this behaviour on hosts with
larger memory usage?


> Can you reproduce the behaviour? Would it be possible to get a recon_alloc
> snaphot during and after load?
>

I'm sharing recon_alloc snapshots during & after load for the following:

1. erts_alloc defaults (lmbcs 5120) [2]
2. -MBlmbcs 512 [3]

I've also captured a during & after load screenshots of the 2
configurations running side-by-side (left is erts_alloc defaults (lmbcs 5120),
right is -MBlmbcs 512) [4].

While our initial configuration used a few more flags, -MHas ageffcbf -MBas
ageffcbf -MHlmbcs 512 -MBlmbcs 512, I've kept things as simple as possible
on this run, and only used -MBlmbcs 512

Have you seen https://github.com/erlang/otp/pull/1790 that was just merged
> to master with the accompanying blog post: http://blog.erlang.org/M
> emory-instrumentation-in-OTP-21/?
>

I haven't, thank you for sharing. We are waiting on Elixir #6611 before we
can test against OTP 21.0-rc1 [5].


> I assume that you have tried ageffcaobf?
>

Yes, we have tried all allocation strategies. ageffcbf resulted in
"spikier" CPU and dirty mem writeback, but also sharper drops in dirty mem
writeback. Under load, ageffcbf had 1% lower RSS usage, and 2.5% lower
unused memory than ageffcaobf. After load however, ageffcbf had 5% lower
RSS usage & 4% lower unused memory. In conclusion, ageffcbf proved the best
out of all allocation strategies.

Here is a side-by-side comparison of -MBas ageffcaobf -MBlmbcs 512 (left)
vs -MBas ageffcbf -MBlmbcs 512 (right) [6], and the relevant recon_alloc
snapshots [7].

Thank you Lukas for helping out with this, Gerhard.

[1] https://github.com/rabbitmq/rabbitmq-server/issues/1223 &
https://github.com/rabbitmq/rabbitmq-server/pull/1259#issuecomment-308428057
- the entire PR context is valuable and relevant
[2] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-MBlmbcs_5120.recon_alloc.snapshot &
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-
MBlmbcs_5120.recon_alloc.snapshot
[3] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-MBlmbcs_512.recon_alloc.snapshot &
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-
MBlmbcs_512.recon_alloc.snapshot
[4] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-MBlmbcs_5120-vs-MBlmbcs_512.png &
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-
MBlmbcs_5120-vs-MBlmbcs_512.png
[5] *https://github.com/elixir-lang/elixir/issues/6611#issuecomment-386208496
<https://github.com/elixir-lang/elixir/issues/6611#issuecomment-386208496>*
[6] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot &
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/after-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot +
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot &
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/after-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot
[7] https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-MBas_ageffcaobf-MBlmbcs_512-vs-
MBas_ageffcbf-MBlmbcs_512.png & https://s3-eu-west-1.
amazonaws.com/rabbitmq-share/memory-allocators/after-MBas_
ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png +
https://s3-eu-west-1.amazonaws.com/rabbitmq-share/
memory-allocators/during-cpu-MBas_ageffcaobf-MBlmbcs_512-
vs-MBas_ageffcbf-MBlmbcs_512.png & https://s3-eu-west-1.
amazonaws.com/rabbitmq-share/memory-allocators/during-
memory-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180503/57827f35/attachment.htm>