[erlang-questions] Hidden Memory Hog

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

List,

I'm at my wits end (they're short, no doubt, but still). I have a VM
running with several OTP applications that we've written. We see, over
the course of the day, memory consumption go up until its taking ~30%
system memory and 300-500MB of swap. Iterating over all processes and
checking their total_heap_size, no single process was greater than ~1
MB, with ~300 processes runnning. Forcing garbage collection on all
processes released maybe 15MB of system memory.

erlang:memory() lists total memory around 18-20MB; ets tables were all
minimal, no dets tables. No processes has a message_queue_len > 0. No
processes that we maintain use the process dictionary.

I started a second VM, migrating our OTP apps to the second VM, one at a
time, hoping that stopping one would cause a release of lots of memory
on the first VM. No such luck; it released a couple more MBs but still a
large presence and no change in swap usage. I then stopped everything I
could until only stdlib and kernel were running on the first VM, but to
no avail.

Each application has 1-3 long running gen_servers that listen on an AMQP
queue and spawn worker processes to handle messages. They carry almost
no state. Some of the handlers will read binary blobs, open a socket,
wait for a connection, then transmit the blobs, closing the socket and
dying afterwards (they timeout after 5 minutes if no connection is
attempted, and are in a supervision tree so I know there aren't a mass
of them lying around). The blobs are between 20KB and 2MB. The rest of
the workers are similarly ephemeral.

What other options are there to find where memory has been allocated? We
use binaries almost exclusively for strings, don't construct large
lists. I've poured over the efficiency guide looking for ideas of where
we've gone astray but am coming up blank so far.

The system is idle; the only processes should be (and are, as far as I
can tell) the gen_servers meant to run for a long time.

Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:8] [kernel-poll:true]

Any ideas are welcome,

James

- -- 
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:james@REDACTED
tel: 415.886.7905
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJONeXJAAoJENc77s1OYoGgx+YIALQzjHIEgfTMUcoYzB9zJ6yd
tGT/J/CccADk/ErTGkpMMSG1PYPa8jzlaIaPw8tOtzKnGWb6iFQGahsPNexKwIKM
t9TwSELRBQeF9SMKBKh/tD1DzIMT47oBpf8uUYTfXy3Ww/ofDEhZ1vKNNkefb3ub
TENEEThDP5UC0L43HYwWp82yIKMSXJvmQUG/2WVvVF7oYXrTd2EPOXtn38+/th5u
pi+j6zFfYaAVZ4ZBijBNJOcePZK7vyw7h/mWQOP+8r5wkETexxaafeX677WIskYu
QMGqp6aABfDl4W4rmFArR3MY47pO0SHjd9MH4JgkhQsbU6OvLFPryr7unq4qQek=
=2WoE
-----END PGP SIGNATURE-----