[erlang-questions] Garbage Collection, BEAM memory and Erlang memory

[Roberto Ostinelli]

On Fri, Jan 23, 2015 at 6:03 PM, Lukas Larsson <garazdawi@REDACTED> wrote:

>
> With a recon_alloc:memory(usage, current) value of 65 - 70%, this
> difference is expected. What is happening is that the memory allocators of
> the VM is not finding any good slots to put new heaps in, so it keeps
> requesting more memory from the OS. erlang:memory reports used memory,
> which is not the same as the memory requested from the OS.
> recon_alloc:memory(allocated, current) reports something very close to what
> has actually been requested from the OS, which is why this value is much
> closer to what you see with top.
>
> It would be interesting to see if you get the same results if you start
> the VM with "+Muacul 0", or if you get better utlization if you use that.
> If you want to know what it is that you are disabling you can read this:
> https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/CarrierMigration.md
> .
>
> Lukas
>

Hi again,
I had the time to try the +Muacul 0 option you suggested. It actually works
pretty well since the RAM usage ratio (erlang / VM) improves considerably.

Unfortunately both with or without the option I am getting the same
problem: the machine seems stable, but at a certain moment *within 30
seconds* the VM memory usage suddenly increases from 11GB to 15GB until it
crashes.

I don't get it because the erlang usage is *very* stable now, I always get
this kind of data:

1> [{K,V / math:pow(1024,3)} || {K,V} <- erlang:memory()].
[{total,8.803221724927425},
 {processes,7.404303453862667},
 {processes_used,7.404133327305317},
 {system,1.3989182710647583},
 {atom,0.0015203068032860756},
 {atom_used,0.0015129167586565018},
 {binary,0.9653719812631607},
 {code,0.049161157570779324},
 {ets,0.007966354489326477}]

I see consistent total, process and binary usage. Unfortunately the ratio
falls:

2> recon_alloc:memory(usage, current).
0.7353255106318904

..after a while:

3> recon_alloc:memory(usage, current).
0.5630988225908702

Why is the VM so eager on memory if the underlying erlang usage is stable?

Is there anything I can do? I honestly don't know where else to look.

   - Binaries are optimized (checked with +bin_opt_info).
   - Erlang reported memory for total, process and binary is linear.
   - I'm using some gimmicks like fullsweep_after 10 as a system flag.
   - I hibernate the long living TCP connections (which is where the
   problem comes from, since I ran tests on short lived connections and had no
   issues).

Any help would be greatly appreciated.

Best,
r.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150127/87d2dda7/attachment.htm>