<div dir="ltr">Hi Lukas,<div class="gmail_extra"><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Why do you not use erlang:memory() as the base for whether you can accept more messages? Having a low memory utilisation is not bad in itself, unless of course some other program on the same machine needs the memory.</div></div></div></div></blockquote><div><br></div><div>We used to use <font face="monospace, monospace" style="background-color:rgb(238,238,238)">erlang:memory()</font>, but we've learned that it doesn't work well in practice [1]. Linux OOM will take action based on RSS, not Erlang allocated memory.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Looking at the used memory under load and after, at peak the allocated memory is 1510 MB and then after it is 577 MB. So about 2/3rds of the allocated memory was returned to the OS. While this is not perfect, it is not terrible either. Reducing it further may not be easy.<br></div></div></div></div></blockquote><div><br></div><div>Your observation is true and accurate. It's also true and accurate that out of 577MB allocated, 300MB is used & 277MB is unused, meaning that almost half of the allocated memory is not used.</div><div><br></div><div>I understand that it may not be easy to reduce the unused memory, but all I'm thinking is that whilst this unused memory might seem small in this particular scenario, what happens when the Erlang VM has 60GB allocated?</div><div><br></div><div>Would it help if we can show the impact of this behaviour on hosts with larger memory usage?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div>Can you reproduce the behaviour? Would it be possible to get a recon_alloc snaphot during and after load?<br></div></span></div></div></div></blockquote><div><br></div><div>I'm sharing <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">recon_alloc snapshots </span>during & after load for the following:</div><div><br></div><div>1. erts_alloc defaults (<font face="monospace, monospace" style="background-color:rgb(238,238,238)">lmbcs 5120</font>) [2]</div><div>2. <span style="background-color:rgb(238,238,238)">-</span><font face="monospace, monospace" style="background-color:rgb(238,238,238)">MBlmbcs 512</font> [3]</div><div> </div><div>I've also captured a during & after load screenshots of the 2 configurations running side-by-side (left is erts_alloc defaults (<font face="monospace, monospace" style="background-color:rgb(238,238,238)">lmbcs 5120</font>), right is <span style="color:rgb(34,34,34);font-family:monospace,monospace;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(238,238,238);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-MBlmbcs 512</span>) [4].</div><div><br></div><div>While our initial configuration used a few more flags, <span style="color:rgb(80,0,80);font-family:monospace,monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(238,238,238);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-MHas ageffcbf -MBas ageffcbf -MHlmbcs 512 -MBlmbcs 512</span>, I've kept things as simple as possible on this run, and only used <font face="monospace, monospace" style="background-color:rgb(238,238,238)">-MBlmbcs 512</font></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div></div></span><div>Have you seen <a href="https://github.com/erlang/otp/pull/1790" target="_blank">https://github.com/erlang<wbr>/otp/pull/1790</a> that was just merged to master with the accompanying blog post: <a href="http://blog.erlang.org/Memory-instrumentation-in-OTP-21/" target="_blank">http://blog.erlang.org/M<wbr>emory-instrumentation-in-OTP-2<wbr>1/</a>?<br></div></div></div></div></blockquote><div><br></div><div>I haven't, thank you for sharing. We are waiting on Elixir #6611 before we can test against OTP 21.0-rc1 [5].</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>I assume that you have tried ageffcaobf?</div></div></div></div></blockquote><div><br></div><div>Yes, we have tried all allocation strategies. <font face="monospace, monospace" style="background-color:rgb(238,238,238)">ageffcbf</font> resulted in "spikier" CPU and dirty mem writeback, but also sharper drops in dirty mem writeback. Under load, <font face="monospace, monospace" style="background-color:rgb(238,238,238)">ageffcbf</font> had 1% lower RSS usage, and 2.5% lower unused memory than <span style="color:rgb(34,34,34);font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;background-color:rgb(238,238,238)"><font face="monospace, monospace">ageffcaobf</font></span>. After load however, <font face="monospace, monospace" style="background-color:rgb(238,238,238)">ageffcbf</font> had 5% lower RSS usage & 4% lower unused memory. In conclusion, <font face="monospace, monospace" style="background-color:rgb(238,238,238)">ageffcbf</font> proved the best out of all allocation strategies.</div><div><br></div><div>Here is a side-by-side comparison of <span style="background-color:rgb(238,238,238)"><font face="monospace, monospace">-MBas </font></span><font face="monospace, monospace" style="background-color:rgb(238,238,238)">ageffcaobf -MBlmbcs 512</font> (left) vs <font face="monospace, monospace" style="background-color:rgb(238,238,238)">-MBas </font><font face="monospace, monospace" style="background-color:rgb(238,238,238)">ageffcbf -MBlmbcs 512</font> (right) [6], and the relevant recon_alloc snapshots [7].</div><div><br></div><div>Thank you Lukas for helping out with this, Gerhard.</div><div><br></div><div>[1] <a href="https://github.com/rabbitmq/rabbitmq-server/issues/1223" target="_blank">https://github.com/rabbitm<wbr>q/rabbitmq-server/issues/1223</a> & <a href="https://github.com/rabbitmq/rabbitmq-server/pull/1259#issuecomment-308428057" target="_blank">https://github.com/rabbitmq/<wbr>rabbitmq-server/pull/1259#issu<wbr>ecomment-308428057</a> - the entire PR context is valuable and relevant</div><div>[2] <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-MBlmbcs_5120.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-<wbr>MBlmbcs_5120.recon_alloc.<wbr>snapshot</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-MBlmbcs_5120.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/after-<wbr>MBlmbcs_5120.recon_alloc.<wbr>snapshot</a></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">[3] <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-MBlmbcs_512.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-<wbr>MBlmbcs_512.recon_alloc.<wbr>snapshot</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-MBlmbcs_512.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/after-<wbr>MBlmbcs_512.recon_alloc.<wbr>snapshot</a></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">[4] <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-MBlmbcs_5120-vs-MBlmbcs_512.png" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-<wbr>MBlmbcs_5120-vs-MBlmbcs_512.<wbr>png</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-MBlmbcs_5120-vs-MBlmbcs_512.png" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/after-<wbr>MBlmbcs_5120-vs-MBlmbcs_512.<wbr>png</a></span></span></span></span></div><div><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px;text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">[5] </span><font color="#1155cc"><u><a href="https://github.com/elixir-lang/elixir/issues/6611#issuecomment-386208496" target="_blank">https://github.com/elixir-<wbr>lang/elixir/issues/6611#issuec<wbr>omment-386208496</a></u></font><br></span></span></span></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">[6] <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-MBas_<wbr>ageffcaobf-MBlmbcs_512.recon_<wbr>alloc.snapshot</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-MBas_ageffcaobf-MBlmbcs_512.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/after-MBas_<wbr>ageffcaobf-MBlmbcs_512.recon_<wbr>alloc.snapshot</a> + <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-MBas_<wbr>ageffcbf-MBlmbcs-512.recon_<wbr>alloc.snapshot</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-MBas_ageffcbf-MBlmbcs-512.recon_alloc.snapshot" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/after-MBas_<wbr>ageffcbf-MBlmbcs-512.recon_<wbr>alloc.snapshot</a></span></span></span></span></span></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">[7] <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-MBas_<wbr>ageffcaobf-MBlmbcs_512-vs-<wbr>MBas_ageffcbf-MBlmbcs_512.png</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/after-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/after-MBas_<wbr>ageffcaobf-MBlmbcs_512-vs-<wbr>MBas_ageffcbf-MBlmbcs_512.png</a> + <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-cpu-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-cpu-<wbr>MBas_ageffcaobf-MBlmbcs_512-<wbr>vs-MBas_ageffcbf-MBlmbcs_512.<wbr>png</a> & <a href="https://s3-eu-west-1.amazonaws.com/rabbitmq-share/memory-allocators/during-memory-MBas_ageffcaobf-MBlmbcs_512-vs-MBas_ageffcbf-MBlmbcs_512.png" target="_blank">https://s3-eu-west-1.<wbr>amazonaws.com/rabbitmq-share/<wbr>memory-allocators/during-<wbr>memory-MBas_ageffcaobf-<wbr>MBlmbcs_512-vs-MBas_ageffcbf-<wbr>MBlmbcs_512.png</a></span></span></span></span></span></span></span></div><div><br></div></div></div></div>