[erlang-questions] Hidden Memory Hog

Wed Aug 3 20:35:31 CEST 2011

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So I started a VM with the +Mim true flag and wondered if folks can help
narrow down where I should be looking for possible offenders?

Using this in the shell:

f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted =
lists:reverse(lists:keysort(3,Mem)).
[{heap,140578906112088,41084720,<0.5051.56>},
 {old_heap,140578980069464,32867800,<0.5051.56>},
 {old_heap,140579035463768,10770152,<0.8797.57>},
 {heap,140578978496600,1571344,<0.8797.57>},
 {heap,140579077124184,971144,<0.25.0>},
 {old_heap,140579095326808,971144,<0.455.0>},
 {old_heap,140579078697048,971144,<0.25.0>},
 {old_heap,140579090284632,600200,<0.404.0>},
 {timer_wheel,140579109355368,524288,undefined},
 {port_tab,140579107569760,466944,undefined},
 {code,140579086084496,326762,<0.25.0>},
 {scheduler_data,140579110159688,287680,undefined},
 {proc_tab,140579109892896,262144,undefined},
 {code,140579070881888,251787,<0.25.0>},
 {code,140579106199176,239377,<0.0.0>},
 {old_heap,140579074121920,229256,<0.396.0>},
 {code,140579082245832,193718,<0.25.0>},
 {code,140579082439600,187723,<0.25.0>},
 {code,140579075601296,183984,<0.25.0>},
 {code,140579085875512,183888,<0.25.0>},
 |...]

The first four lines are the shell and a process spawned by the shell.

If I run:

lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted).
112277446

Is that number valid? Should I be looking at lines with type heap or
old_heap only?

The memory output from /proc/PID/status:

VmPeak:	  728864 kB
VmSize:	  335188 kB
VmLck:	       0 kB
VmHWM:	  421760 kB
VmRSS:	  250352 kB
VmData:	  306400 kB
VmStk:	     136 kB
VmExe:	    1800 kB
VmLib:	    3748 kB
VmPTE:	     680 kB
VmSwap:	    1520 kB

Any direction for where to continue searching?

Thanks for the help so far,

James

On 08/01/2011 10:32 AM, Mihai Balea wrote:
> That looks pretty benign.  I suspect the OS reported memory usage for the beam process is vastly higher, correct?
> Do you run any native code? NIFs, linked-in drivers? Maybe there's a leak in there.
> 
> On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote:
> 
> No, no outliers.
> 
> 1> erlang:memory().
> [{total,29393336},
> {processes,18646408},
> {processes_used,18560824},
> {system,10746928},
> {atom,658865},
> {atom_used,652857},
> {binary,1059104},
> {code,6608997},
> {ets,434848}]
> 
> Unless I'm reading that output incorrectly?
> 
> On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote:
>>>> What does erlang:memory() show? Binaries, processes or ets uses that
>>>> much memory?
>>>>
>>>> 2011/8/1, James Aimonetti <james@REDACTED>:
>>>> List,
>>>>
>>>> I'm at my wits end (they're short, no doubt, but still). I have a VM
>>>> running with several OTP applications that we've written. We see, over
>>>> the course of the day, memory consumption go up until its taking ~30%
>>>> system memory and 300-500MB of swap. Iterating over all processes and
>>>> checking their total_heap_size, no single process was greater than ~1
>>>> MB, with ~300 processes runnning. Forcing garbage collection on all
>>>> processes released maybe 15MB of system memory.
>>>>
>>>> erlang:memory() lists total memory around 18-20MB; ets tables were all
>>>> minimal, no dets tables. No processes has a message_queue_len > 0. No
>>>> processes that we maintain use the process dictionary.
>>>>
>>>> I started a second VM, migrating our OTP apps to the second VM, one at a
>>>> time, hoping that stopping one would cause a release of lots of memory
>>>> on the first VM. No such luck; it released a couple more MBs but still a
>>>> large presence and no change in swap usage. I then stopped everything I
>>>> could until only stdlib and kernel were running on the first VM, but to
>>>> no avail.
>>>>
>>>> Each application has 1-3 long running gen_servers that listen on an AMQP
>>>> queue and spawn worker processes to handle messages. They carry almost
>>>> no state. Some of the handlers will read binary blobs, open a socket,
>>>> wait for a connection, then transmit the blobs, closing the socket and
>>>> dying afterwards (they timeout after 5 minutes if no connection is
>>>> attempted, and are in a supervision tree so I know there aren't a mass
>>>> of them lying around). The blobs are between 20KB and 2MB. The rest of
>>>> the workers are similarly ephemeral.
>>>>
>>>> What other options are there to find where memory has been allocated? We
>>>> use binaries almost exclusively for strings, don't construct large
>>>> lists. I've poured over the efficiency guide looking for ideas of where
>>>> we've gone astray but am coming up blank so far.
>>>>
>>>> The system is idle; the only processes should be (and are, as far as I
>>>> can tell) the gen_servers meant to run for a long time.
>>>>
>>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]
>>>> [async-threads:8] [kernel-poll:true]
>>>>
>>>> Any ideas are welcome,
>>>>
>>>> James
>>>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED
>>>> http://erlang.org/mailman/listinfo/erlang-questions
> 
_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://erlang.org/mailman/listinfo/erlang-questions

- -- 
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:james@REDACTED
tel: 415.886.7905
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOOZTzAAoJENc77s1OYoGgNSQH+QGcD5WDBSJp+z6RCjAqcRS+
TM0fxxLolnWN9wdeje/v91ZA+G+usz/LbR0LhctwD7gJk29Ni56CO2cNP1osw5/c
VWMSvxllDqn6ZmqAGl+h0lpYH9ZLWiZNZ1f4BjihHLjyglhdAKdRYfbQSzZsilQA
Zu2AcfBvSQ564ByOEnU4ZEqrVaY9+EnI7WOjfd9HMHG2GBPgpDJauUGWfF21Qglf
w7FESB/Gv6TMUHgdBEpBXmi1A1XgJFlQh8kK0xz6rNEp7GbHHrzk9Nl/644HzwGJ
jsMpmBf6r+NXDwlR4Brd7eIdoIcnJ7DFIGEfUHFS73zkFAGxQdOoDCDGgnTSwSc=
=jp4S
-----END PGP SIGNATURE-----