[erlang-questions] ll_alloc reallocate crash

[Mikael Pettersson]

mliu writes:
 > We have a simple tcp server spawning many processes that manage
 > accepted connections. these connections are long-lived, with long
 > periods of idle waits (5 min). While stress-testing the server, we
 > came across a strange memory-related crash (dump below). At the time
 > of the crash, we have around 150k connections, and roughly 4 processes
 > per conn (~600K). Server has 8Gb memory.
 > 
 > Hoping someone has a clue of where we should look, within erlang or
 > maybe os configuration.
...
 > Here's the crash dump:
 > --
 > Slogan: ll_alloc: Cannot reallocate 2033104 bytes of memory (of type
 > "fd_status").
 > System version: Erlang R13B01 (erts-5.7.2) [source] [smp:8:8] [rq:8]
 > [async-threads:0] [hipe] [kernel-poll:true]
 > Compiled: Wed Aug 26 15:33:27 2009
 > Atoms: 9193
 > =memory
 > total: 3396483592
 > processes: 2538096436
 > processes_used: 2538092308
 > system: 858387156
 > atom: 416385
 > atom_used: 395052
 > binary: 248350576
 > code: 3168606
 > ets: 209338444
...

I can't analyze crash dumps, but this does look like a normal
out-of-memory situation, though whether you ran out of virtual
address space or physical memory I can't tell. Things to try:
- build the Erlang VM as a 64-bit executable, otherwise it will
  be limited to about 3GB of virtual address space
- do you have swap on the machine? if not, maybe you should