[erlang-questions] Hidden Memory Hog

[Attila Rajmund Nohl]

What does erlang:memory() show? Binaries, processes or ets uses that
much memory?

2011/8/1, James Aimonetti <james@REDACTED>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> List,
>
> I'm at my wits end (they're short, no doubt, but still). I have a VM
> running with several OTP applications that we've written. We see, over
> the course of the day, memory consumption go up until its taking ~30%
> system memory and 300-500MB of swap. Iterating over all processes and
> checking their total_heap_size, no single process was greater than ~1
> MB, with ~300 processes runnning. Forcing garbage collection on all
> processes released maybe 15MB of system memory.
>
> erlang:memory() lists total memory around 18-20MB; ets tables were all
> minimal, no dets tables. No processes has a message_queue_len > 0. No
> processes that we maintain use the process dictionary.
>
> I started a second VM, migrating our OTP apps to the second VM, one at a
> time, hoping that stopping one would cause a release of lots of memory
> on the first VM. No such luck; it released a couple more MBs but still a
> large presence and no change in swap usage. I then stopped everything I
> could until only stdlib and kernel were running on the first VM, but to
> no avail.
>
> Each application has 1-3 long running gen_servers that listen on an AMQP
> queue and spawn worker processes to handle messages. They carry almost
> no state. Some of the handlers will read binary blobs, open a socket,
> wait for a connection, then transmit the blobs, closing the socket and
> dying afterwards (they timeout after 5 minutes if no connection is
> attempted, and are in a supervision tree so I know there aren't a mass
> of them lying around). The blobs are between 20KB and 2MB. The rest of
> the workers are similarly ephemeral.
>
> What other options are there to find where memory has been allocated? We
> use binaries almost exclusively for strings, don't construct large
> lists. I've poured over the efficiency guide looking for ideas of where
> we've gone astray but am coming up blank so far.
>
> The system is idle; the only processes should be (and are, as far as I
> can tell) the gen_servers meant to run for a long time.
>
> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]
> [async-threads:8] [kernel-poll:true]
>
> Any ideas are welcome,
>
> James
>
> - --
> James Aimonetti
> Distributed Systems Engineer / DJ MC_
>
> 2600hz | http://2600hz.com
> sip:james@REDACTED
> tel: 415.886.7905
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJONeXJAAoJENc77s1OYoGgx+YIALQzjHIEgfTMUcoYzB9zJ6yd
> tGT/J/CccADk/ErTGkpMMSG1PYPa8jzlaIaPw8tOtzKnGWb6iFQGahsPNexKwIKM
> t9TwSELRBQeF9SMKBKh/tD1DzIMT47oBpf8uUYTfXy3Ww/ofDEhZ1vKNNkefb3ub
> TENEEThDP5UC0L43HYwWp82yIKMSXJvmQUG/2WVvVF7oYXrTd2EPOXtn38+/th5u
> pi+j6zFfYaAVZ4ZBijBNJOcePZK7vyw7h/mWQOP+8r5wkETexxaafeX677WIskYu
> QMGqp6aABfDl4W4rmFArR3MY47pO0SHjd9MH4JgkhQsbU6OvLFPryr7unq4qQek=
> =2WoE
> -----END PGP SIGNATURE-----
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>