Discrepancy between OS memory and erlang virtual memory?

[Dominic Letz]

Great that it helped! Yes I went for OTP 24. If you don't want to switch
yet you can also so just disable the affected Erlang VM carrier.

https://erlang.org/doc/man/erts_alloc.html
E.g. to disable the binary alloc carrier add `+MBe false` to your erlang
start args.

Best


Hitesh Kishorbhai Vaghani <hitesh.v@REDACTED> schrieb am Do., 23. Sep.
2021, 10:23:

> Dominic, Thanks for your scripts . It has helped to identify issues
> with {binary_alloc,3140268032}, All the missing memory is allocated into
> binary_alloc.
> We do using binary_to_term on 1 to 4 MB blob variables. Which version of
> OTP have you migrated to? We are also planning to migrate OTP21 to OTP24.
>
> Roger,
> We are not using NIFs. We using heavy concurrent processing to reduce
> response time. how to check the default allocator?
>
> *Regards,*
>
> *Hitesh Vaghani*
>
>
> On Thu, Sep 23, 2021 at 1:06 PM Roger Lipscombe <roger@REDACTED>
> wrote:
>
>> Yes, recon has useful functions for reporting on fragmentation.
>>
>> Question: are you doing anything with NIFs? We had a problem a few
>> years ago where the default glibc allocator in Ubuntu 14/16, when used
>> in a heavily multi-threaded way (i.e. when being called from Erlang),
>> caused a *lot* of fragmentation, and it's hard to track down.
>> Switching to a more modern allocator (we chose jemalloc) basically
>> resolved the problem immediately.
>>
>> On Thu, 23 Sept 2021 at 08:13, Dominic Letz <dominic@REDACTED> wrote:
>> >
>> > The erlang vm has it's own memory allocators (carriers) they usually
>> create a small memory overhead. I've seen them consuming exorbitant amounts
>> of memory before, likely because of memory fragmentation. Their usage is
>> not reported in the erlang virtual memory stats by default but you can
>> check them explicitly.
>> >
>> > When fighting with memory fragmentation I created this script to show
>> the current carrier overhead: (waste)
>> https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc
>> >
>> > My solution in the end was to upgrade OTP to a newer version. And it
>> made the crazy overheads go away.
>> >
>> >  Cheers!
>> >
>> > Max Lapshin <max.lapshin@REDACTED> schrieb am Do., 23. Sep. 2021,
>> 08:42:
>> >>
>> >> You show a strange graphic without any numbers. all other commands are
>> also cutted.
>> >>
>> >> Also part of your email is rather unclear: you write to a public list
>> that your email is confidential. Bad idea.
>> >>
>> >> About memory:
>> >>
>> >> 1) use recon in your production. recon_alloc will help to collect big
>> amount of information from erlang runtime
>> >> 2) take a look at /proc/3975367/maps, sometimes it helps to find leaks
>> or better interpret numbers.
>> >>
>> >> What you see is a mismatch between different ways to calculate memory.
>> Usually it is ok to live with such a difference.
>> >>
>> >>
>>
>
>
>
>
> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
> contents of this e-mail and any attachments are confidential and intended
> for the named recipient(s) only.E-mail transmission is not guaranteed to be
> secure or error-free as information could be intercepted, corrupted,lost,
> destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents(with or without referred errors)
> shall therefore not attach any liability on the originator or redBus.com.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the views or opinions of redBus.com.
> Any form of reproduction, dissemination, copying, disclosure,
> modification,distribution and / or publication of this message without the
> prior written consent of authorized representative of redbus.
> <http://redbus.in/>com is strictly prohibited. If you have received this
> email in error please delete it and notify the sender immediately.Before
> opening any email and/or attachments, please check them for viruses and
> other defects.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/8fd5382b/attachment.htm>