Discrepancy between OS memory and erlang virtual memory?

[Roger Lipscombe]

Yes, recon has useful functions for reporting on fragmentation.

Question: are you doing anything with NIFs? We had a problem a few
years ago where the default glibc allocator in Ubuntu 14/16, when used
in a heavily multi-threaded way (i.e. when being called from Erlang),
caused a *lot* of fragmentation, and it's hard to track down.
Switching to a more modern allocator (we chose jemalloc) basically
resolved the problem immediately.

On Thu, 23 Sept 2021 at 08:13, Dominic Letz <dominic@REDACTED> wrote:
>
> The erlang vm has it's own memory allocators (carriers) they usually create a small memory overhead. I've seen them consuming exorbitant amounts of memory before, likely because of memory fragmentation. Their usage is not reported in the erlang virtual memory stats by default but you can check them explicitly.
>
> When fighting with memory fragmentation I created this script to show the current carrier overhead: (waste) https://gist.github.com/dominicletz/615e4b89b9e6f2059b2520ed9adac5dc
>
> My solution in the end was to upgrade OTP to a newer version. And it made the crazy overheads go away.
>
>  Cheers!
>
> Max Lapshin <max.lapshin@REDACTED> schrieb am Do., 23. Sep. 2021, 08:42:
>>
>> You show a strange graphic without any numbers. all other commands are also cutted.
>>
>> Also part of your email is rather unclear: you write to a public list that your email is confidential. Bad idea.
>>
>> About memory:
>>
>> 1) use recon in your production. recon_alloc will help to collect big amount of information from erlang runtime
>> 2) take a look at /proc/3975367/maps, sometimes it helps to find leaks or better interpret numbers.
>>
>> What you see is a mismatch between different ways to calculate memory. Usually it is ok to live with such a difference.
>>
>>