memory used by Erlang VM

[Jesper Louis Andersen]

It is expected your RSS value is lower than what Erlang reports.

Erlang requests virtual memory from the operating system (Linux). But it is
mapped into the process on-demand. RSS is the resident set size, which is
the currently physical mapped pages. As Erlang hits pages it hasn't used
before, a kernel trap is generated and it maps in that page to to the
process.

For example, lets say we allocate a large array to hold processes, but we
are only using a small smidgen of that array. Then there are many virtually
mapped pages, but we aren't really "using" the memory yet. It is taken
on-demand. This on-demand method is smart because it lowers memory
pressure. Rather than having to do all the work up front, we can amortize
the work over the course of the program running, which is much more
efficient. If the application requests 1 gigabyte of memory say, and we
want to give it right away, we must zero one gigabyte of memory. This takes
time. But on-demand, we can keep a background process for zeroing memory
and allocate it as it is needed, among other things.

Where this can create a problem is if you have several processes all
wanting lots of memory, and you have less memory in the machine than what
they've got promised by the kernel. Then, if they all starts wanting
memory, you should expect to see various trouble.

As Erlang runs more, I would expect the RSS to go up as it populates more
of the memory space.

On Fri, Jan 31, 2020 at 8:23 PM Vyacheslav Levytskyy <v.levytskyy@REDACTED>
wrote:

> It's linux (ubuntu). I read /proc/[pid]/statm, take RSS number of pages
> (the 2nd value in the line of that file) and calculate memory used by
> Erlang VM as RSS x page size (read as "getconf PAGESIZE").
> On 31.01.2020 19:59, Jesper Louis Andersen wrote:
>
> When you say /proc/pid, what are you looking at specifically in there? It
> is a bit different depending on which Unix you run on, so a simple
> example will help a lot.
>
> In particular, my early guess is going to be virtual memory vs physical
> RSS mapping. The former can be much higher than the latter. Especially in
> system such as Linux, which allow overcommitting memory.
>
> On Fri, Jan 31, 2020 at 7:22 PM Vyacheslav Levytskyy <
> v.levytskyy@REDACTED> wrote:
>
>> Hello,
>>
>> I wonder why memory used by Erlang VM as reported by the kernel via the
>> /proc/pid differs from erlang:memory(total). In my current configuration
>> I observe realistic response from erlang:memory(total) and much lower
>> values from the /proc/pid.
>>
>> I'm not surprised by the difference itself, but rather by the fact that
>> the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
>> but it looks like RabbitMQ is using the /proc/pid approach by default,
>> proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
>> available options of Erlang VM memory consumption calculation strategy.
>>
>> Does anybody have insights of what and why is going on with those
>> calculations of memory used by Erlang VM? Is it possible to select one
>> strategy beforehand for my Erlang app, or I must measure on each new
>> configuration what looks more precise? Should I compare and change the
>> strategy during run-time, or after I selected a strategy once for my
>> configuration I can be sure that selected approach always better than
>> other two?
>>
>> Thank you,
>> Vyacheslav
>>
>
>
> --
> J.
>
>

-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200131/a1baa60c/attachment.htm>