<div dir="ltr"><div><br></div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px">Hi All,</div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px">We did disable +MBe false but then it was difficult to track VM memory and internal leaks.</div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px">So we updated to OTP 24 and we have come across a different issue. Now, as soon as we start our service in production it is occupying 11 GB+ for binary allocation. When we checked system memory it was just 1 GB or less. Is there any significant changes in OTP 24 at VM level? I read the release but nothing was mentioned there.</div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><b>Memory stack:recon_alloc:memory(allocated_types), (Service is just started and not handling any request, CPU: 1%)</b></div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><div>[{binary_alloc,12754513920},</div><div> {driver_alloc,11567104},</div><div> {eheap_alloc,284672000},</div><div> {ets_alloc,1599770624},</div><div> {fix_alloc,11042816},</div><div> {ll_alloc,118751232},</div><div> {sl_alloc,294912},</div><div> {std_alloc,5537792},</div><div> {temp_alloc,1179648}]</div></div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><br></div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><b>Top:</b></div><div style="color:rgba(0,0,0,0.87);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><div>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div>3167429 omegaus+ 20 0 12.70g 0.2g 9880 S 379.0 20.7 3411:19 beam.smp </div><div><br></div></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><b>Regards,</b><br></p><p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:12pt;font-family:"Times New Roman",serif">Hitesh Vaghani</span></b><span style="font-size:12pt;font-family:"Times New Roman",serif"> </span></p></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 23, 2021 at 5:42 PM Hitesh Kishorbhai Vaghani <<a href="mailto:hitesh.v@redbus.com">hitesh.v@redbus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Dominic,<div><br></div><div>We will try both your options and update about our findings in the next few days.Thanks for all the help. Since these are production systems, I can't experiment on weekends.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 23, 2021 at 5:00 PM Dominic Letz <<a href="mailto:dominic@diode.io" target="_blank">dominic@diode.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><span style="font-family:sans-serif;font-size:12.8px">Great that it helped! Yes I went for OTP 24. If you don't want to switch yet you can also so just disable the affected Erlang VM carrier.</span><div dir="auto" style="font-family:sans-serif;font-size:12.8px"><br></div><div dir="auto" style="font-family:sans-serif;font-size:12.8px"><a href="https://erlang.org/doc/man/erts_alloc.html" style="text-decoration-line:none;color:rgb(66,133,244)" target="_blank">https://erlang.org/doc/man/erts_alloc.html</a><br></div><div dir="auto" style="font-family:sans-serif;font-size:12.8px">E.g. to disable the binary alloc carrier add `+MBe false` to your erlang start args.</div><div dir="auto" style="font-family:sans-serif;font-size:12.8px"><br></div><div dir="auto" style="font-family:sans-serif;font-size:12.8px">Best</div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Hitesh Kishorbhai Vaghani <<a href="mailto:hitesh.v@redbus.com" target="_blank">hitesh.v@redbus.com</a>> schrieb am Do., 23. Sep. 2021, 10:23:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dominic, Thanks for your scripts . It has helped to identify issues with {binary_alloc,3140268032}, All the missing memory is allocated into binary_alloc.<div>We do using binary_to_term on 1 to 4 MB blob variables. Which version of OTP have you migrated to? We are also planning to migrate OTP21 to OTP24.</div><div><br></div><div>Roger,</div><div>We are not using NIFs. We using heavy concurrent processing to reduce response time. how to check the default allocator? </div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><b>Regards,</b><br></p><p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:12pt;font-family:"Times New Roman",serif">Hitesh Vaghani</span></b><span style="font-size:12pt;font-family:"Times New Roman",serif"> </span></p></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 23, 2021 at 1:06 PM Roger Lipscombe <<a href="mailto:roger@differentpla.net" rel="noreferrer" target="_blank">roger@differentpla.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Yes, recon has useful functions for reporting on fragmentation.<br>
<br>
Question: are you doing anything with NIFs? We had a problem a few<br>
years ago where the default glibc allocator in Ubuntu 14/16, when used<br>
in a heavily multi-threaded way (i.e. when being called from Erlang),<br>
caused a *lot* of fragmentation, and it's hard to track down.<br>
Switching to a more modern allocator (we chose jemalloc) basically<br>
resolved the problem immediately.<br>
<br>
On Thu, 23 Sept 2021 at 08:13, Dominic Letz <<a href="mailto:dominic@diode.io" rel="noreferrer" target="_blank">dominic@diode.io</a>> wrote:<br>
><br>
> The erlang vm has it's own memory allocators (carriers) they usually create a small memory overhead. I've seen them consuming exorbitant amounts of memory before, likely because of memory fragmentation. Their usage is not reported in the erlang virtual memory stats by default but you can check them explicitly.<br>
><br>
> When fighting with memory fragmentation I created this script to show the current carrier overhead: (waste) <a href="https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc" rel="noreferrer noreferrer" target="_blank">https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc</a><br>
><br>
> My solution in the end was to upgrade OTP to a newer version. And it made the crazy overheads go away.<br>
><br>
> Cheers!<br>
><br>
> Max Lapshin <<a href="mailto:max.lapshin@gmail.com" rel="noreferrer" target="_blank">max.lapshin@gmail.com</a>> schrieb am Do., 23. Sep. 2021, 08:42:<br>
>><br>
>> You show a strange graphic without any numbers. all other commands are also cutted.<br>
>><br>
>> Also part of your email is rather unclear: you write to a public list that your email is confidential. Bad idea.<br>
>><br>
>> About memory:<br>
>><br>
>> 1) use recon in your production. recon_alloc will help to collect big amount of information from erlang runtime<br>
>> 2) take a look at /proc/3975367/maps, sometimes it helps to find leaks or better interpret numbers.<br>
>><br>
>> What you see is a mismatch between different ways to calculate memory. Usually it is ok to live with such a difference.<br>
>><br>
>><br>
</blockquote></div>
<br>
<font size="3"><b style="color:rgb(0,0,0);font-family:"Times New Roman";font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium;font-weight:normal"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">::DISCLAIMER::</span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">----------------------------------------------------------------------------------------------------------------------------------------------------</span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of</span><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><a href="http://redbus.in/" rel="noreferrer" target="_blank"> redbus.</a>com</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.</span></b></font></blockquote></div></div></div>
</blockquote></div>
</blockquote></div>
<br>
<font size="3"><b style="color:rgb(0,0,0);font-family:'Times New Roman';font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium;font-weight:normal"><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">::DISCLAIMER::</span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>----------------------------</span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></span><br><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of</span><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><a href="http://redbus.in/" target="_blank"> redbus.</a>com</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.</span></b></font>