[erlang-questions] Relationship between erlang:memory/1, ets:info/2 and erlang:process_info/2

[Rickard Green]

erlang:memory() and 'top' reports information about two different things.

erlang:memory() reports the amount of memory that the runtime system has dynamically requested from the erts internal memory allocators. Memory allocators will satisfy these request by placing memory blocks in larger memory areas which we call carriers. There will both be memory used for allocator internal data structures as well as free unused memory in these carriers which is not accounted for by erlang:memory(), and shouldn't be accounted for by erlang:memory().

'top' reports all memory mapped by the emulator process. This includes the carriers used by the erts internal memory allocators, but also statically allocated memory, code of the emulator, etc...

When the emulator has grown large, memory carriers will normally (but not necessarily always) make up the major part of the memory reported by 'top' and other similar tools. I've attached a module, called memory, that can be used in order to sum up the sizes of all used carriers. The memory allocators may however keep unused carriers a while in order to try to reuse them. These unused carriers are not accounted for by the memory-module. Also note that this is a sum of sub results acquired at different (but close) points in time.

It should be possible to use the memory-module on R10B and upwards. Note that it is more or less untested. I've only smoke tested it on R15B02.

Regards,
Rickard Green, Erlang/OTP, Ericsson AB.

On Sep 22, 2012, at 10:18 AM, Anton Lebedevich wrote:

> There could be even more inconsistensies between memory consumed by
> erlang vm and statistics reported by memory().
> 
> top output
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 15863 root      20   0  721m 559m 3316 S    0 14.1   3:47.58 beam.smp
> 
> (node@REDACTED)1> memory().
> [{total,103534784},
> {processes,18226104},
> {processes_used,14971560},
> {system,85308680},
> {atom,2247985},
> {atom_used,2227049},
> {binary,33598272},
> {code,25217445},
> {ets,1857024}]
> 
> Total in memory() output is about 100Mb but real memory usage of
> erlang vm (R14B04) is ~600Mb
> 
> On Fri, Sep 21, 2012 at 8:50 PM, Simon MacMullen <simon@REDACTED> wrote:
>> I'm trying to write something that will account for memory use on a RabbitMQ
>> system. To do this I'm trying to match up the numbers I get out of
>> erlang:memory(ets) and ets:info(T, memory), and erlang:memory(processes) and
>> erlang:process_info/2.
>> 
>> And it doesn't make a huge amount of sense to me. I don't expect things to
>> match up perfectly, of course there will be some memory unaccounted for, but
>> the ETS numbers in particular can be quite off:
>> 
>> 1> erlang:memory(ets).
>> 2120291872
>> 
>> 2> lists:sum([ets:info(T, memory) || T <- ets:all()]) *
>> erlang:system_info(wordsize).
>> 1874292776
>> 
>> So I have ~2GB of ETS memory, and ~230MB unaccounted for. Should I expect
>> these to match up closer?
>> 
>> And then...
>> 
>> 3> erlang:memory(processes).
>> 1519425156
>> 
>> 4> lists:sum([element(2, process_info(P, memory)) || P <- processes()]).
>> 1552852008
>> 
>> So for processes there is less in total than if you add them all up. Is some
>> memory shared? (Are we counting binaries and atoms in there as well?)
>> 
>> Is there a way to get this sort of information in a more consistent manner?
>> 
>> Cheers, Simon
>> 
>> --
>> Simon MacMullen
>> RabbitMQ, VMware
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory.erl
Type: application/octet-stream
Size: 2151 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120924/3d664bd3/attachment.obj>
-------------- next part --------------