[erlang-questions] discrepancy of memory usage figures

[Jesper Louis Andersen]

On Thu, Jun 23, 2011 at 05:39, Andy W. Song <wsongcn@REDACTED> wrote:

> The figures in red are provided htop, others are output of memory() call
> from erlang shell. Please look at the total and erlang residence memory.
> When there is no connection, these two are roughly same, with 100K
> connections, residence memory is a little larger than total, with 200K
> connections, residence memory is almost double the total.
> Can anybody explain?

The first thing I would analyze in this situation is memory
fragmentation obtained from the underlying memory allocation. If the
memory fragments, then surely you can see stuff like this happening
where your resident set size will be larger. I would definitely start
out by setting the spotlight on the underlying allocator or the
allocation pattern because as you see, there is no discrepancy if you
ask the Erlang VM about its view on the matter. What speaks against
fragmentation is the very young life of the program. Fragmentation
problems usually only show up in long-running programs where pointers
are set in stone (making compaction impossible).

Another option is that there is some auxiliary data structure in the
Erlang VM which are grown up in between the 100K and 200K connection
count. If the structure is accessed randomly, then surely all the
Virtual Memory pages will get allocated and assigned to the process,
increasing the RSS of the program.

It is also interesting to look for what happens if you simply ignore
the program for a bit of time. Perhaps the allocated memory is not
released right away, but over time.

Apart from my ideas, I am afraid you will need an Erlang VM expert
with more intrinsic knowledge than me.


-- 
J.