Discrepancy between OS memory and erlang virtual memory?

Thu Sep 23 14:12:37 CEST 2021

Hi Dominic,

We will try both your options and update about our findings in the next few
days.Thanks for all the help. Since these are production systems, I can't
experiment on weekends.

On Thu, Sep 23, 2021 at 5:00 PM Dominic Letz <dominic@REDACTED> wrote:

> Great that it helped! Yes I went for OTP 24. If you don't want to switch
> yet you can also so just disable the affected Erlang VM carrier.
>
> https://erlang.org/doc/man/erts_alloc.html
> E.g. to disable the binary alloc carrier add `+MBe false` to your erlang
> start args.
>
> Best
>
>
> Hitesh Kishorbhai Vaghani <hitesh.v@REDACTED> schrieb am Do., 23. Sep.
> 2021, 10:23:
>
>> Dominic, Thanks for your scripts . It has helped to identify issues
>> with {binary_alloc,3140268032}, All the missing memory is allocated into
>> binary_alloc.
>> We do using binary_to_term on 1 to 4 MB blob variables. Which version of
>> OTP have you migrated to? We are also planning to migrate OTP21 to OTP24.
>>
>> Roger,
>> We are not using NIFs. We using heavy concurrent processing to reduce
>> response time. how to check the default allocator?
>>
>> *Regards,*
>>
>> *Hitesh Vaghani*
>>
>>
>> On Thu, Sep 23, 2021 at 1:06 PM Roger Lipscombe <roger@REDACTED>
>> wrote:
>>
>>> Yes, recon has useful functions for reporting on fragmentation.
>>>
>>> Question: are you doing anything with NIFs? We had a problem a few
>>> years ago where the default glibc allocator in Ubuntu 14/16, when used
>>> in a heavily multi-threaded way (i.e. when being called from Erlang),
>>> caused a *lot* of fragmentation, and it's hard to track down.
>>> Switching to a more modern allocator (we chose jemalloc) basically
>>> resolved the problem immediately.
>>>
>>> On Thu, 23 Sept 2021 at 08:13, Dominic Letz <dominic@REDACTED> wrote:
>>> >
>>> > The erlang vm has it's own memory allocators (carriers) they usually
>>> create a small memory overhead. I've seen them consuming exorbitant amounts
>>> of memory before, likely because of memory fragmentation. Their usage is
>>> not reported in the erlang virtual memory stats by default but you can
>>> check them explicitly.
>>> >
>>> > When fighting with memory fragmentation I created this script to show
>>> the current carrier overhead: (waste)
>>> https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc
>>> >
>>> > My solution in the end was to upgrade OTP to a newer version. And it
>>> made the crazy overheads go away.
>>> >
>>> >  Cheers!
>>> >
>>> > Max Lapshin <max.lapshin@REDACTED> schrieb am Do., 23. Sep. 2021,
>>> 08:42:
>>> >>
>>> >> You show a strange graphic without any numbers. all other commands
>>> are also cutted.
>>> >>
>>> >> Also part of your email is rather unclear: you write to a public list
>>> that your email is confidential. Bad idea.
>>> >>
>>> >> About memory:
>>> >>
>>> >> 1) use recon in your production. recon_alloc will help to collect big
>>> amount of information from erlang runtime
>>> >> 2) take a look at /proc/3975367/maps, sometimes it helps to find
>>> leaks or better interpret numbers.
>>> >>
>>> >> What you see is a mismatch between different ways to calculate
>>> memory. Usually it is ok to live with such a difference.
>>> >>
>>> >>
>>>
>>
>>
>>
>>
>> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
>> contents of this e-mail and any attachments are confidential and intended
>> for the named recipient(s) only.E-mail transmission is not guaranteed to be
>> secure or error-free as information could be intercepted, corrupted,lost,
>> destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents(with or without referred errors)
>> shall therefore not attach any liability on the originator or redBus.com.
>> Views or opinions, if any, presented in this email are solely those of the
>> author and may not necessarily reflect the views or opinions of redBus.com.
>> Any form of reproduction, dissemination, copying, disclosure,
>> modification,distribution and / or publication of this message without the
>> prior written consent of authorized representative of redbus.
>> <http://redbus.in/>com is strictly prohibited. If you have received this
>> email in error please delete it and notify the sender immediately.Before
>> opening any email and/or attachments, please check them for viruses and
>> other defects.*
>
>

-- 
*::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and 
intended for the named recipient(s) only.E-mail transmission is not 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted,lost, destroyed, arrive late or incomplete, or may contain 
viruses in transmission. The e mail and its contents(with or without 
referred errors) shall therefore not attach any liability on the originator 
or redBus.com. Views or opinions, if any, presented in this email are 
solely those of the author and may not necessarily reflect the views or 
opinions of redBus.com. Any form of reproduction, dissemination, copying, 
disclosure, modification,distribution and / or publication of this message 
without the prior written consent of authorized representative of redbus. 
<http://redbus.in/>com is strictly prohibited. If you have received this 
email in error please delete it and notify the sender immediately.Before 
opening any email and/or attachments, please check them for viruses and 
other defects.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/2f6ccbce/attachment.htm>