Discrepancy between OS memory and erlang virtual memory?

[Hitesh Kishorbhai Vaghani]

Dominic, Thanks for your scripts . It has helped to identify issues
with {binary_alloc,3140268032}, All the missing memory is allocated into
binary_alloc.
We do using binary_to_term on 1 to 4 MB blob variables. Which version of
OTP have you migrated to? We are also planning to migrate OTP21 to OTP24.

Roger,
We are not using NIFs. We using heavy concurrent processing to reduce
response time. how to check the default allocator?

*Regards,*

*Hitesh Vaghani*


On Thu, Sep 23, 2021 at 1:06 PM Roger Lipscombe <roger@REDACTED>
wrote:

> Yes, recon has useful functions for reporting on fragmentation.
>
> Question: are you doing anything with NIFs? We had a problem a few
> years ago where the default glibc allocator in Ubuntu 14/16, when used
> in a heavily multi-threaded way (i.e. when being called from Erlang),
> caused a *lot* of fragmentation, and it's hard to track down.
> Switching to a more modern allocator (we chose jemalloc) basically
> resolved the problem immediately.
>
> On Thu, 23 Sept 2021 at 08:13, Dominic Letz <dominic@REDACTED> wrote:
> >
> > The erlang vm has it's own memory allocators (carriers) they usually
> create a small memory overhead. I've seen them consuming exorbitant amounts
> of memory before, likely because of memory fragmentation. Their usage is
> not reported in the erlang virtual memory stats by default but you can
> check them explicitly.
> >
> > When fighting with memory fragmentation I created this script to show
> the current carrier overhead: (waste)
> https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc
> >
> > My solution in the end was to upgrade OTP to a newer version. And it
> made the crazy overheads go away.
> >
> >  Cheers!
> >
> > Max Lapshin <max.lapshin@REDACTED> schrieb am Do., 23. Sep. 2021,
> 08:42:
> >>
> >> You show a strange graphic without any numbers. all other commands are
> also cutted.
> >>
> >> Also part of your email is rather unclear: you write to a public list
> that your email is confidential. Bad idea.
> >>
> >> About memory:
> >>
> >> 1) use recon in your production. recon_alloc will help to collect big
> amount of information from erlang runtime
> >> 2) take a look at /proc/3975367/maps, sometimes it helps to find leaks
> or better interpret numbers.
> >>
> >> What you see is a mismatch between different ways to calculate memory.
> Usually it is ok to live with such a difference.
> >>
> >>
>

-- 
*::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and 
intended for the named recipient(s) only.E-mail transmission is not 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted,lost, destroyed, arrive late or incomplete, or may contain 
viruses in transmission. The e mail and its contents(with or without 
referred errors) shall therefore not attach any liability on the originator 
or redBus.com. Views or opinions, if any, presented in this email are 
solely those of the author and may not necessarily reflect the views or 
opinions of redBus.com. Any form of reproduction, dissemination, copying, 
disclosure, modification,distribution and / or publication of this message 
without the prior written consent of authorized representative of redbus. 
<http://redbus.in/>com is strictly prohibited. If you have received this 
email in error please delete it and notify the sender immediately.Before 
opening any email and/or attachments, please check them for viruses and 
other defects.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/f7b67083/attachment.htm>