[erlang-questions] Problem with Beam process not freeing memory

[Tino Breddin]

Hi list,

I'm experiencing issues with a couple of Beam nodes where I see a huge gap
between the node's reported memory usage and the underlying Linux Kernel's
view.

This is using R15B01.

As a start an application in such a node stores a lot of tuples (containing
atoms and binary data) in ETS tables. That proceeds until a point where
memory usage is 70% (6GB) of the available memory. At that point
erlang:memory() and top (or /proc/PID/status) agree roughly on the memory
usage. Then an internal cleanup task is performed, which clears obsolete
records from the ETS tables. Afterwards, erlang:memory() reports an
expected low value of roughly 60MB memory usage. (This includes binary
data). However, the kernel still reports the high memory usage values (both
VmRss and VmTotal) for the node. The kernel's usage view will stay stable
until the ETS tables are filled to a point where the real memory usage
exceeds the kernel's view, then the kernel reported usage will grow as well.

Now having checked the node in some details I'm wondering what causes this
difference between the BEAM's view and the Kernel's view on memory usage. I
have 2 ideas which I'm checking right now.

(1) Not GC'ed binaries: Could it be that binary data is not GC'ed because
the original dispatcher process which it was passed through before being
stored in an ETS table is still alive. Thus there is still some reference
to it? However, this would not explain why erlang:memory() reports a very
low value for used memory for binaries.

(2) low-level memory leak: Some driver or NIF leaking memory, which would
obviously not be reported by erlang:memory(). However, then it surprises me
that the Kernel's view stays stable while the BEAM's actual memory usage is
still below the Kernel's view. It should be continuously growing in this
case imho.

I'd appreciate if anyone has some more insight or experience with such a
behaviour, while I'm further digging into this.

Cheers,
Tino
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130916/a92221dd/attachment.htm>