[erlang-questions] Problem with Beam process not freeing memory

[Masklinn]

On 2013-09-16, at 13:40 , Tino Breddin wrote:
> 
> (1) Not GC'ed binaries: Could it be that binary data is not GC'ed because
> the original dispatcher process which it was passed through before being
> stored in an ETS table is still alive. Thus there is still some reference
> to it? However, this would not explain why erlang:memory() reports a very
> low value for used memory for binaries.
> 
> (2) low-level memory leak: Some driver or NIF leaking memory, which would
> obviously not be reported by erlang:memory(). However, then it surprises me
> that the Kernel's view stays stable while the BEAM's actual memory usage is
> still below the Kernel's view. It should be continuously growing in this
> case imho.
> 
> I'd appreciate if anyone has some more insight or experience with such a
> behaviour, while I'm further digging into this.

I have no info, but I'd suggest #3: the runtime does not release memory
back to the OS under the assumption that the high-water mark probably
isn't a unique event; or there is some delay between the moment where
memory is released to the runtime and where memory is released by the
runtime to the kernel (IIRC CPython has such a behavior with multi-stage
allocations, internal pools and freelists[-1] and that's a fairly recent
evolution, before 2006 CPython *never* released memory to the kernel[0]).

[-1] this means memory released by a Python program may or may not be
     immediately free()'d back to the kernel, and due to release being
     predicated upon arena emptying[0] and ignoring freelists it will
     not happen for all internally released memory, at least not without
     an explicit full GC and maybe even with that
[0] http://bugs.python.org/issue1123430