[erlang-questions] Garbage collection tuning for long living processes

[Jonas Boberg]

Hi,

We have an Erlang server with tens of thousands of medium/long living
processes. Each of these processes handles a TCP connection. The
Erlang VM garbage collector does not size the heap of these processes
to our liking.  This seems to be due to the memory allocation pattern:
* The processes initially (during connection setup) need a lot of
memory -  sometimes up to several megabytes. The VM (correctly) grows
the heap of the process, typically we see a heap_size of 514229 or
319408.
* After the connection setup, the processes need relatively little
memory, typically less than one kilobyte.
* At random intervals after connection setup, the process needs a
medium amount of memory (100s of kilobytes) for a short time, and then
goes back to the low allocation pattern.

The heap is never down-sized. Looking at the GC code in VM, I think
this is because a GC is not run until a allocation requires more
memory than available on the heap. It takes hours for this to happen,
since the heap initial grows to several megabytes, and little memory
is allocated.

Things tried:
* Setting the spawn option {fullsweep_after, 0}  - does not improve
the situation. Since garbage collection doesn't occur, neither does a
fullsweep.
* Hibernating the process after every handled client request. While
this minimizes the heap size, it increases CPU usage significantly.
The thing is that the processes aren't going inactive, so they are
constantly waked up from hibernation as messages are passed to them.
We don't really have a good criteria of when to hibernate.

Are there any other tuning parameters in the VM that might help us?

Regards
Jonas