Discrepancy between OS memory and erlang virtual memory?

Tue Nov 30 07:46:40 CET 2021

Hi All,
We did disable +MBe false but then it was difficult to track VM memory and
internal leaks.
So we updated to OTP 24 and we have come across a different issue. Now, as
soon as we start our service in production it is occupying 11 GB+ for
binary allocation. When we checked system memory it was just 1 GB or less.
Is there any significant changes in OTP 24 at VM level? I read the release
but nothing was mentioned there.
*Memory stack:recon_alloc:memory(allocated_types), (Service is just started
and not handling any request, CPU: 1%)*
[{binary_alloc,12754513920},
 {driver_alloc,11567104},
 {eheap_alloc,284672000},
 {ets_alloc,1599770624},
 {fix_alloc,11042816},
 {ll_alloc,118751232},
 {sl_alloc,294912},
 {std_alloc,5537792},
 {temp_alloc,1179648}]

*Top:*
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

3167429 omegaus+  20   0   12.70g   0.2g   9880 S 379.0  20.7   3411:19
beam.smp

*Regards,*

*Hitesh Vaghani*

On Thu, Sep 23, 2021 at 5:42 PM Hitesh Kishorbhai Vaghani <
hitesh.v@REDACTED> wrote:

> Hi Dominic,
>
> We will try both your options and update about our findings in the next
> few days.Thanks for all the help. Since these are production systems, I
> can't experiment on weekends.
>
> On Thu, Sep 23, 2021 at 5:00 PM Dominic Letz <dominic@REDACTED> wrote:
>
>> Great that it helped! Yes I went for OTP 24. If you don't want to switch
>> yet you can also so just disable the affected Erlang VM carrier.
>>
>> https://erlang.org/doc/man/erts_alloc.html
>> E.g. to disable the binary alloc carrier add `+MBe false` to your erlang
>> start args.
>>
>> Best
>>
>>
>> Hitesh Kishorbhai Vaghani <hitesh.v@REDACTED> schrieb am Do., 23. Sep.
>> 2021, 10:23:
>>
>>> Dominic, Thanks for your scripts . It has helped to identify issues
>>> with {binary_alloc,3140268032}, All the missing memory is allocated into
>>> binary_alloc.
>>> We do using binary_to_term on 1 to 4 MB blob variables. Which version of
>>> OTP have you migrated to? We are also planning to migrate OTP21 to OTP24.
>>>
>>> Roger,
>>> We are not using NIFs. We using heavy concurrent processing to reduce
>>> response time. how to check the default allocator?
>>>
>>> *Regards,*
>>>
>>> *Hitesh Vaghani*
>>>
>>>
>>> On Thu, Sep 23, 2021 at 1:06 PM Roger Lipscombe <roger@REDACTED>
>>> wrote:
>>>
>>>> Yes, recon has useful functions for reporting on fragmentation.
>>>>
>>>> Question: are you doing anything with NIFs? We had a problem a few
>>>> years ago where the default glibc allocator in Ubuntu 14/16, when used
>>>> in a heavily multi-threaded way (i.e. when being called from Erlang),
>>>> caused a *lot* of fragmentation, and it's hard to track down.
>>>> Switching to a more modern allocator (we chose jemalloc) basically
>>>> resolved the problem immediately.
>>>>
>>>> On Thu, 23 Sept 2021 at 08:13, Dominic Letz <dominic@REDACTED> wrote:
>>>> >
>>>> > The erlang vm has it's own memory allocators (carriers) they usually
>>>> create a small memory overhead. I've seen them consuming exorbitant amounts
>>>> of memory before, likely because of memory fragmentation. Their usage is
>>>> not reported in the erlang virtual memory stats by default but you can
>>>> check them explicitly.
>>>> >
>>>> > When fighting with memory fragmentation I created this script to show
>>>> the current carrier overhead: (waste)
>>>> https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc
>>>> >
>>>> > My solution in the end was to upgrade OTP to a newer version. And it
>>>> made the crazy overheads go away.
>>>> >
>>>> >  Cheers!
>>>> >
>>>> > Max Lapshin <max.lapshin@REDACTED> schrieb am Do., 23. Sep. 2021,
>>>> 08:42:
>>>> >>
>>>> >> You show a strange graphic without any numbers. all other commands
>>>> are also cutted.
>>>> >>
>>>> >> Also part of your email is rather unclear: you write to a public
>>>> list that your email is confidential. Bad idea.
>>>> >>
>>>> >> About memory:
>>>> >>
>>>> >> 1) use recon in your production. recon_alloc will help to collect
>>>> big amount of information from erlang runtime
>>>> >> 2) take a look at /proc/3975367/maps, sometimes it helps to find
>>>> leaks or better interpret numbers.
>>>> >>
>>>> >> What you see is a mismatch between different ways to calculate
>>>> memory. Usually it is ok to live with such a difference.
>>>> >>
>>>> >>
>>>>
>>>
>>>
>>>
>>>
>>> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
>>> contents of this e-mail and any attachments are confidential and intended
>>> for the named recipient(s) only.E-mail transmission is not guaranteed to be
>>> secure or error-free as information could be intercepted, corrupted,lost,
>>> destroyed, arrive late or incomplete, or may contain viruses in
>>> transmission. The e mail and its contents(with or without referred errors)
>>> shall therefore not attach any liability on the originator or redBus.com.
>>> Views or opinions, if any, presented in this email are solely those of the
>>> author and may not necessarily reflect the views or opinions of redBus.com.
>>> Any form of reproduction, dissemination, copying, disclosure,
>>> modification,distribution and / or publication of this message without the
>>> prior written consent of authorized representative of redbus.
>>> <http://redbus.in/>com is strictly prohibited. If you have received this
>>> email in error please delete it and notify the sender immediately.Before
>>> opening any email and/or attachments, please check them for viruses and
>>> other defects.*
>>
>>

-- 
*::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and 
intended for the named recipient(s) only.E-mail transmission is not 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted,lost, destroyed, arrive late or incomplete, or may contain 
viruses in transmission. The e mail and its contents(with or without 
referred errors) shall therefore not attach any liability on the originator 
or redBus.com. Views or opinions, if any, presented in this email are 
solely those of the author and may not necessarily reflect the views or 
opinions of redBus.com. Any form of reproduction, dissemination, copying, 
disclosure, modification,distribution and / or publication of this message 
without the prior written consent of authorized representative of redbus. 
<http://redbus.in/>com is strictly prohibited. If you have received this 
email in error please delete it and notify the sender immediately.Before 
opening any email and/or attachments, please check them for viruses and 
other defects.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20211130/0a483452/attachment.htm>