[erlang-questions] Hidden Memory Hog

James Aimonetti james@REDACTED
Wed Aug 3 20:35:31 CEST 2011

Hash: SHA1

So I started a VM with the +Mim true flag and wondered if folks can help
narrow down where I should be looking for possible offenders?

Using this in the shell:

f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted =

The first four lines are the shell and a process spawned by the shell.

If I run:

lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted).

Is that number valid? Should I be looking at lines with type heap or
old_heap only?

The memory output from /proc/PID/status:

VmPeak:	  728864 kB
VmSize:	  335188 kB
VmLck:	       0 kB
VmHWM:	  421760 kB
VmRSS:	  250352 kB
VmData:	  306400 kB
VmStk:	     136 kB
VmExe:	    1800 kB
VmLib:	    3748 kB
VmPTE:	     680 kB
VmSwap:	    1520 kB

Any direction for where to continue searching?

Thanks for the help so far,


On 08/01/2011 10:32 AM, Mihai Balea wrote:
> That looks pretty benign.  I suspect the OS reported memory usage for the beam process is vastly higher, correct?
> Do you run any native code? NIFs, linked-in drivers? Maybe there's a leak in there.
> On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote:
> No, no outliers.
> 1> erlang:memory().
> [{total,29393336},
> {processes,18646408},
> {processes_used,18560824},
> {system,10746928},
> {atom,658865},
> {atom_used,652857},
> {binary,1059104},
> {code,6608997},
> {ets,434848}]
> Unless I'm reading that output incorrectly?
> On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote:
>>>> What does erlang:memory() show? Binaries, processes or ets uses that
>>>> much memory?
>>>> 2011/8/1, James Aimonetti <james@REDACTED>:
>>>> List,
>>>> I'm at my wits end (they're short, no doubt, but still). I have a VM
>>>> running with several OTP applications that we've written. We see, over
>>>> the course of the day, memory consumption go up until its taking ~30%
>>>> system memory and 300-500MB of swap. Iterating over all processes and
>>>> checking their total_heap_size, no single process was greater than ~1
>>>> MB, with ~300 processes runnning. Forcing garbage collection on all
>>>> processes released maybe 15MB of system memory.
>>>> erlang:memory() lists total memory around 18-20MB; ets tables were all
>>>> minimal, no dets tables. No processes has a message_queue_len > 0. No
>>>> processes that we maintain use the process dictionary.
>>>> I started a second VM, migrating our OTP apps to the second VM, one at a
>>>> time, hoping that stopping one would cause a release of lots of memory
>>>> on the first VM. No such luck; it released a couple more MBs but still a
>>>> large presence and no change in swap usage. I then stopped everything I
>>>> could until only stdlib and kernel were running on the first VM, but to
>>>> no avail.
>>>> Each application has 1-3 long running gen_servers that listen on an AMQP
>>>> queue and spawn worker processes to handle messages. They carry almost
>>>> no state. Some of the handlers will read binary blobs, open a socket,
>>>> wait for a connection, then transmit the blobs, closing the socket and
>>>> dying afterwards (they timeout after 5 minutes if no connection is
>>>> attempted, and are in a supervision tree so I know there aren't a mass
>>>> of them lying around). The blobs are between 20KB and 2MB. The rest of
>>>> the workers are similarly ephemeral.
>>>> What other options are there to find where memory has been allocated? We
>>>> use binaries almost exclusively for strings, don't construct large
>>>> lists. I've poured over the efficiency guide looking for ideas of where
>>>> we've gone astray but am coming up blank so far.
>>>> The system is idle; the only processes should be (and are, as far as I
>>>> can tell) the gen_servers meant to run for a long time.
>>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]
>>>> [async-threads:8] [kernel-poll:true]
>>>> Any ideas are welcome,
>>>> James
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED
>>>> http://erlang.org/mailman/listinfo/erlang-questions
erlang-questions mailing list

- -- 
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
tel: 415.886.7905
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/


More information about the erlang-questions mailing list