ll_alloc reallocate crash

[mliu]

We have a simple tcp server spawning many processes that manage
accepted connections. these connections are long-lived, with long
periods of idle waits (5 min). While stress-testing the server, we
came across a strange memory-related crash (dump below). At the time
of the crash, we have around 150k connections, and roughly 4 processes
per conn (~600K). Server has 8Gb memory.

Hoping someone has a clue of where we should look, within erlang or
maybe os configuration.

ulimit -a (CentOS)
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71680
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 900000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 71680
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.netdev_max_backlog = 3000

Here's the crash dump:
--
Slogan: ll_alloc: Cannot reallocate 2033104 bytes of memory (of type
"fd_status").
System version: Erlang R13B01 (erts-5.7.2) [source] [smp:8:8] [rq:8]
[async-threads:0] [hipe] [kernel-poll:true]
Compiled: Wed Aug 26 15:33:27 2009
Atoms: 9193
=memory
total: 3396483592
processes: 2538096436
processes_used: 2538092308
system: 858387156
atom: 416385
atom_used: 395052
binary: 248350576
code: 3168606
ets: 209338444
=hash_table:atom_tab
size: 6421
used: 4862
objs: 9193
depth: 7
=index_table:atom_tab
size: 9216
limit: 1048576
entries: 9193
=hash_table:module_code
size: 197
used: 118
objs: 151
depth: 3
=index_table:module_code
size: 1024
limit: 65536
entries: 151
=hash_table:export_list
size: 3203
used: 2324
objs: 4110
depth: 7
=index_table:export_list
size: 5120
limit: 65536
entries: 4110
=hash_table:secondary_export_table
size: 97
used: 0
objs: 0
depth: 0
=hash_table:process_reg
size: 97
used: 53
objs: 74
depth: 3
=hash_table:fun_table
size: 797
used: 516
objs: 807
depth: 6
=hash_table:node_table
size: 5
used: 2
objs: 3
depth: 2
=hash_table:dist_table
size: 5
used: 2
objs: 3
depth: 2