[erlang-questions] Hidden Memory Hog

[Mihai Balea]

That looks pretty benign.  I suspect the OS reported memory usage for the beam process is vastly higher, correct?
Do you run any native code? NIFs, linked-in drivers? Maybe there's a leak in there.

On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> No, no outliers.
> 
> 1> erlang:memory().
> [{total,29393336},
> {processes,18646408},
> {processes_used,18560824},
> {system,10746928},
> {atom,658865},
> {atom_used,652857},
> {binary,1059104},
> {code,6608997},
> {ets,434848}]
> 
> Unless I'm reading that output incorrectly?
> 
> On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote:
>> What does erlang:memory() show? Binaries, processes or ets uses that
>> much memory?
>> 
>> 2011/8/1, James Aimonetti <james@REDACTED>:
>> List,
>> 
>> I'm at my wits end (they're short, no doubt, but still). I have a VM
>> running with several OTP applications that we've written. We see, over
>> the course of the day, memory consumption go up until its taking ~30%
>> system memory and 300-500MB of swap. Iterating over all processes and
>> checking their total_heap_size, no single process was greater than ~1
>> MB, with ~300 processes runnning. Forcing garbage collection on all
>> processes released maybe 15MB of system memory.
>> 
>> erlang:memory() lists total memory around 18-20MB; ets tables were all
>> minimal, no dets tables. No processes has a message_queue_len > 0. No
>> processes that we maintain use the process dictionary.
>> 
>> I started a second VM, migrating our OTP apps to the second VM, one at a
>> time, hoping that stopping one would cause a release of lots of memory
>> on the first VM. No such luck; it released a couple more MBs but still a
>> large presence and no change in swap usage. I then stopped everything I
>> could until only stdlib and kernel were running on the first VM, but to
>> no avail.
>> 
>> Each application has 1-3 long running gen_servers that listen on an AMQP
>> queue and spawn worker processes to handle messages. They carry almost
>> no state. Some of the handlers will read binary blobs, open a socket,
>> wait for a connection, then transmit the blobs, closing the socket and
>> dying afterwards (they timeout after 5 minutes if no connection is
>> attempted, and are in a supervision tree so I know there aren't a mass
>> of them lying around). The blobs are between 20KB and 2MB. The rest of
>> the workers are similarly ephemeral.
>> 
>> What other options are there to find where memory has been allocated? We
>> use binaries almost exclusively for strings, don't construct large
>> lists. I've poured over the efficiency guide looking for ideas of where
>> we've gone astray but am coming up blank so far.
>> 
>> The system is idle; the only processes should be (and are, as far as I
>> can tell) the gen_servers meant to run for a long time.
>> 
>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]
>> [async-threads:8] [kernel-poll:true]
>> 
>> Any ideas are welcome,
>> 
>> James
>> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>>> 
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
> 
> - -- 
> James Aimonetti
> Distributed Systems Engineer / DJ MC_
> 
> 2600hz | http://2600hz.com
> sip:james@REDACTED
> tel: 415.886.7905
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQEcBAEBAgAGBQJONsx1AAoJENc77s1OYoGgviIH/jhqGwz+Y3UoJcxQTAABqiZK
> Ho8o7+TIMKbPqWsnsNChA8S8ERJGiL8e+seuAYWvTo9Qw9tVgaOCgj8A6wOxeLJ1
> SWp2DmjJAkunAZz2g58d02agiykZcAIiOSUZxhY4Z7phjmqLAz5C9UQs4Ih5mCSA
> qpeza+zht3pRqpO+Uo8Vz3OAMK5aRD7GiI7iC6n5nwP2ZBFBHMNaLx1JFNcXq9dF
> jGFdcuoFyZeKnHV40DxKuUcCx9pwyjUr5E4Xq+0gPqdmVEJRuOacwvYk5nlAT+BH
> xKCPmvjpyTrxenBYexpuYgDIBWWpESfNOohoKtXSYd/vTAS/W32f0oLPKraS3I0=
> =yE1n
> -----END PGP SIGNATURE-----
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions