[erlang-questions] Hidden Memory Hog

James Aimonetti james@REDACTED
Mon Sep 16 16:07:40 CEST 2013 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

It ended up being a port driver that wasn't freeing memory in the C
code (an older version of the erlang-syslog lib). But this was two
years ago, so details are fuzzy for me :)

James

On 09/16/2013 12:28 AM, Tino Breddin wrote:
> Hi James,
> 
> I'm seeing similar issues on a cluster installation using R15B01.
> Did you get anywhere and figure out the issue?
> 
> Cheers, Tino
> 
> 
> On Wed, Aug 3, 2011 at 8:35 PM, James Aimonetti <james@REDACTED>
> wrote:
> 
> So I started a VM with the +Mim true flag and wondered if folks can
> help narrow down where I should be looking for possible offenders?
> 
> Using this in the shell:
> 
> f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted = 
> lists:reverse(lists:keysort(3,Mem)). 
> [{heap,140578906112088,41084720,<0.5051.56>}, 
> {old_heap,140578980069464,32867800,<0.5051.56>}, 
> {old_heap,140579035463768,10770152,<0.8797.57>}, 
> {heap,140578978496600,1571344,<0.8797.57>}, 
> {heap,140579077124184,971144,<0.25.0>}, 
> {old_heap,140579095326808,971144,<0.455.0>}, 
> {old_heap,140579078697048,971144,<0.25.0>}, 
> {old_heap,140579090284632,600200,<0.404.0>}, 
> {timer_wheel,140579109355368,524288,undefined}, 
> {port_tab,140579107569760,466944,undefined}, 
> {code,140579086084496,326762,<0.25.0>}, 
> {scheduler_data,140579110159688,287680,undefined}, 
> {proc_tab,140579109892896,262144,undefined}, 
> {code,140579070881888,251787,<0.25.0>}, 
> {code,140579106199176,239377,<0.0.0>}, 
> {old_heap,140579074121920,229256,<0.396.0>}, 
> {code,140579082245832,193718,<0.25.0>}, 
> {code,140579082439600,187723,<0.25.0>}, 
> {code,140579075601296,183984,<0.25.0>}, 
> {code,140579085875512,183888,<0.25.0>}, |...]
> 
> The first four lines are the shell and a process spawned by the
> shell.
> 
> If I run:
> 
> lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted). 
> 112277446
> 
> Is that number valid? Should I be looking at lines with type heap
> or old_heap only?
> 
> The memory output from /proc/PID/status:
> 
> VmPeak:   728864 kB VmSize:   335188 kB VmLck:         0 kB VmHWM:
> 421760 kB VmRSS:    250352 kB VmData:   306400 kB VmStk:       136
> kB VmExe:      1800 kB VmLib:      3748 kB VmPTE:       680 kB 
> VmSwap:     1520 kB
> 
> Any direction for where to continue searching?
> 
> Thanks for the help so far,
> 
> James
> 
> On 08/01/2011 10:32 AM, Mihai Balea wrote:
>>>> That looks pretty benign.  I suspect the OS reported memory
>>>> usage for
> the beam process is vastly higher, correct?
>>>> Do you run any native code? NIFs, linked-in drivers? Maybe
>>>> there's a
> leak in there.
>>>> 
>>>> On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote:
>>>> 
>>>> No, no outliers.
>>>> 
>>>> 1> erlang:memory(). [{total,29393336}, {processes,18646408}, 
>>>> {processes_used,18560824}, {system,10746928}, {atom,658865}, 
>>>> {atom_used,652857}, {binary,1059104}, {code,6608997}, 
>>>> {ets,434848}]
>>>> 
>>>> Unless I'm reading that output incorrectly?
>>>> 
>>>> On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote:
>>>>>>> What does erlang:memory() show? Binaries, processes or
>>>>>>> ets uses that much memory?
>>>>>>> 
>>>>>>> 2011/8/1, James Aimonetti <james@REDACTED>: List,
>>>>>>> 
>>>>>>> I'm at my wits end (they're short, no doubt, but
>>>>>>> still). I have a VM running with several OTP
>>>>>>> applications that we've written. We see, over the
>>>>>>> course of the day, memory consumption go up until its
>>>>>>> taking ~30% system memory and 300-500MB of swap.
>>>>>>> Iterating over all processes and checking their
>>>>>>> total_heap_size, no single process was greater than ~1 
>>>>>>> MB, with ~300 processes runnning. Forcing garbage
>>>>>>> collection on all processes released maybe 15MB of
>>>>>>> system memory.
>>>>>>> 
>>>>>>> erlang:memory() lists total memory around 18-20MB; ets
>>>>>>> tables were all minimal, no dets tables. No processes
>>>>>>> has a message_queue_len > 0. No processes that we
>>>>>>> maintain use the process dictionary.
>>>>>>> 
>>>>>>> I started a second VM, migrating our OTP apps to the
>>>>>>> second VM, one
> at a
>>>>>>> time, hoping that stopping one would cause a release of
>>>>>>> lots of memory on the first VM. No such luck; it
>>>>>>> released a couple more MBs but
> still a
>>>>>>> large presence and no change in swap usage. I then
>>>>>>> stopped everything
> I
>>>>>>> could until only stdlib and kernel were running on the
>>>>>>> first VM, but
> to
>>>>>>> no avail.
>>>>>>> 
>>>>>>> Each application has 1-3 long running gen_servers that
>>>>>>> listen on an
> AMQP
>>>>>>> queue and spawn worker processes to handle messages.
>>>>>>> They carry almost no state. Some of the handlers will
>>>>>>> read binary blobs, open a socket, wait for a
>>>>>>> connection, then transmit the blobs, closing the socket
>>>>>>> and dying afterwards (they timeout after 5 minutes if
>>>>>>> no connection is attempted, and are in a supervision
>>>>>>> tree so I know there aren't a mass of them lying
>>>>>>> around). The blobs are between 20KB and 2MB. The rest
>>>>>>> of the workers are similarly ephemeral.
>>>>>>> 
>>>>>>> What other options are there to find where memory has
>>>>>>> been allocated?
> We
>>>>>>> use binaries almost exclusively for strings, don't
>>>>>>> construct large lists. I've poured over the efficiency
>>>>>>> guide looking for ideas of
> where
>>>>>>> we've gone astray but am coming up blank so far.
>>>>>>> 
>>>>>>> The system is idle; the only processes should be (and
>>>>>>> are, as far as I can tell) the gen_servers meant to run
>>>>>>> for a long time.
>>>>>>> 
>>>>>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4]
>>>>>>> [rq:4] [async-threads:8] [kernel-poll:true]
>>>>>>> 
>>>>>>> Any ideas are welcome,
>>>>>>> 
>>>>>>> James
>>>>>>> 
>>>> _______________________________________________ 
>>>> erlang-questions mailing list erlang-questions@REDACTED 
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>> 
>>>>>>> _______________________________________________ 
>>>>>>> erlang-questions mailing list 
>>>>>>> erlang-questions@REDACTED 
>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>> 
> _______________________________________________ erlang-questions
> mailing list erlang-questions@REDACTED 
> http://erlang.org/mailman/listinfo/erlang-questions
> 
>> _______________________________________________ erlang-questions
>> mailing list erlang-questions@REDACTED 
>> http://erlang.org/mailman/listinfo/erlang-questions
>> 
> 

- -- 
James Aimonetti
Lead Systems Architect
"I thought I fixed that"

2600Hz | http://2600hz.com
sip:james@REDACTED
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlI3EKwACgkQ54NxaUq7OmASeAEAqxRKZyBrbsWUPBCezGiaQeRb
PTDtNXnCEM5lq2gnBAcBALcISObSHxtSVHN/xI/sZ8GDzaUHIrddoZvy2AAG1Stl
=Jxd5
-----END PGP SIGNATURE-----

More information about the erlang-questions mailing list