[erlang-questions] automatic garbage collection when low memo ry?

[Matthias Lang]

Convey Christian J NPRI writes:

 > Two possibilities occur to me:

 > (1) The spawned worker processes are fragmenting the Linux process'
 > heap, so that each time I kick off a new generation of worker
 > processes they're unable to reuse the memory that the previous
 > generation of worker processes had used, or

 > (2) The worker processes return their results by sending a message
 > back to the control processes just before they die.  Perhaps their
 > results require lots of memory to hold.

 > Has anyone seen (1) before?  And regarding (2), is there some way I
 > can calculate the memory usage of an Erlang value (in this case, a
 > list of lists of tuples)?

I have seen fragmentation before, though I'm hazy about the
details. You can re-live it starting here:

  http://www.erlang.org/pipermail/erlang-questions/2002-October/005924.html

though the subject line was a bit unnecessary. Keeping in mind that
the thread is quite a few years old, the main point is that there's
quite a bit of tuning you can do with memory allocators. Fiddling with
the parameters is likely to provide some insight even if it doesn't
solve the problem.

(2) is tricky because the same Erlang term can occupy different
amounts of memory depending on what you've done with it. Message
passing copies everything apart from binaries in a term, so that's a
kind of worst-case. 

size(term_to_binary(Term)) is not a bad estimate.

Matt