[erlang-questions] Hidden Memory Hog

Mon Sep 16 09:28:03 CEST 2013

Hi James,

I'm seeing similar issues on a cluster installation using R15B01. Did you
get anywhere and figure out the issue?

Cheers,
Tino

On Wed, Aug 3, 2011 at 8:35 PM, James Aimonetti <james@REDACTED> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> So I started a VM with the +Mim true flag and wondered if folks can help
> narrow down where I should be looking for possible offenders?
>
> Using this in the shell:
>
> f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted =
> lists:reverse(lists:keysort(3,Mem)).
> [{heap,140578906112088,41084720,<0.5051.56>},
>  {old_heap,140578980069464,32867800,<0.5051.56>},
>  {old_heap,140579035463768,10770152,<0.8797.57>},
>  {heap,140578978496600,1571344,<0.8797.57>},
>  {heap,140579077124184,971144,<0.25.0>},
>  {old_heap,140579095326808,971144,<0.455.0>},
>  {old_heap,140579078697048,971144,<0.25.0>},
>  {old_heap,140579090284632,600200,<0.404.0>},
>  {timer_wheel,140579109355368,524288,undefined},
>  {port_tab,140579107569760,466944,undefined},
>  {code,140579086084496,326762,<0.25.0>},
>  {scheduler_data,140579110159688,287680,undefined},
>  {proc_tab,140579109892896,262144,undefined},
>  {code,140579070881888,251787,<0.25.0>},
>  {code,140579106199176,239377,<0.0.0>},
>  {old_heap,140579074121920,229256,<0.396.0>},
>  {code,140579082245832,193718,<0.25.0>},
>  {code,140579082439600,187723,<0.25.0>},
>  {code,140579075601296,183984,<0.25.0>},
>  {code,140579085875512,183888,<0.25.0>},
>  |...]
>
> The first four lines are the shell and a process spawned by the shell.
>
> If I run:
>
> lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted).
> 112277446
>
> Is that number valid? Should I be looking at lines with type heap or
> old_heap only?
>
> The memory output from /proc/PID/status:
>
> VmPeak:   728864 kB
> VmSize:   335188 kB
> VmLck:         0 kB
> VmHWM:    421760 kB
> VmRSS:    250352 kB
> VmData:   306400 kB
> VmStk:       136 kB
> VmExe:      1800 kB
> VmLib:      3748 kB
> VmPTE:       680 kB
> VmSwap:     1520 kB
>
> Any direction for where to continue searching?
>
> Thanks for the help so far,
>
> James
>
> On 08/01/2011 10:32 AM, Mihai Balea wrote:
> > That looks pretty benign.  I suspect the OS reported memory usage for
> the beam process is vastly higher, correct?
> > Do you run any native code? NIFs, linked-in drivers? Maybe there's a
> leak in there.
> >
> > On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote:
> >
> > No, no outliers.
> >
> > 1> erlang:memory().
> > [{total,29393336},
> > {processes,18646408},
> > {processes_used,18560824},
> > {system,10746928},
> > {atom,658865},
> > {atom_used,652857},
> > {binary,1059104},
> > {code,6608997},
> > {ets,434848}]
> >
> > Unless I'm reading that output incorrectly?
> >
> > On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote:
> >>>> What does erlang:memory() show? Binaries, processes or ets uses that
> >>>> much memory?
> >>>>
> >>>> 2011/8/1, James Aimonetti <james@REDACTED>:
> >>>> List,
> >>>>
> >>>> I'm at my wits end (they're short, no doubt, but still). I have a VM
> >>>> running with several OTP applications that we've written. We see, over
> >>>> the course of the day, memory consumption go up until its taking ~30%
> >>>> system memory and 300-500MB of swap. Iterating over all processes and
> >>>> checking their total_heap_size, no single process was greater than ~1
> >>>> MB, with ~300 processes runnning. Forcing garbage collection on all
> >>>> processes released maybe 15MB of system memory.
> >>>>
> >>>> erlang:memory() lists total memory around 18-20MB; ets tables were all
> >>>> minimal, no dets tables. No processes has a message_queue_len > 0. No
> >>>> processes that we maintain use the process dictionary.
> >>>>
> >>>> I started a second VM, migrating our OTP apps to the second VM, one
> at a
> >>>> time, hoping that stopping one would cause a release of lots of memory
> >>>> on the first VM. No such luck; it released a couple more MBs but
> still a
> >>>> large presence and no change in swap usage. I then stopped everything
> I
> >>>> could until only stdlib and kernel were running on the first VM, but
> to
> >>>> no avail.
> >>>>
> >>>> Each application has 1-3 long running gen_servers that listen on an
> AMQP
> >>>> queue and spawn worker processes to handle messages. They carry almost
> >>>> no state. Some of the handlers will read binary blobs, open a socket,
> >>>> wait for a connection, then transmit the blobs, closing the socket and
> >>>> dying afterwards (they timeout after 5 minutes if no connection is
> >>>> attempted, and are in a supervision tree so I know there aren't a mass
> >>>> of them lying around). The blobs are between 20KB and 2MB. The rest of
> >>>> the workers are similarly ephemeral.
> >>>>
> >>>> What other options are there to find where memory has been allocated?
> We
> >>>> use binaries almost exclusively for strings, don't construct large
> >>>> lists. I've poured over the efficiency guide looking for ideas of
> where
> >>>> we've gone astray but am coming up blank so far.
> >>>>
> >>>> The system is idle; the only processes should be (and are, as far as I
> >>>> can tell) the gen_servers meant to run for a long time.
> >>>>
> >>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]
> >>>> [async-threads:8] [kernel-poll:true]
> >>>>
> >>>> Any ideas are welcome,
> >>>>
> >>>> James
> >>>>
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
> >>>>>
> >>>> _______________________________________________
> >>>> erlang-questions mailing list
> >>>> erlang-questions@REDACTED
> >>>> http://erlang.org/mailman/listinfo/erlang-questions
> >
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
> - --
> James Aimonetti
> Distributed Systems Engineer / DJ MC_
>
> 2600hz | http://2600hz.com
> sip:james@REDACTED
> tel: 415.886.7905
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJOOZTzAAoJENc77s1OYoGgNSQH+QGcD5WDBSJp+z6RCjAqcRS+
> TM0fxxLolnWN9wdeje/v91ZA+G+usz/LbR0LhctwD7gJk29Ni56CO2cNP1osw5/c
> VWMSvxllDqn6ZmqAGl+h0lpYH9ZLWiZNZ1f4BjihHLjyglhdAKdRYfbQSzZsilQA
> Zu2AcfBvSQ564ByOEnU4ZEqrVaY9+EnI7WOjfd9HMHG2GBPgpDJauUGWfF21Qglf
> w7FESB/Gv6TMUHgdBEpBXmi1A1XgJFlQh8kK0xz6rNEp7GbHHrzk9Nl/644HzwGJ
> jsMpmBf6r+NXDwlR4Brd7eIdoIcnJ7DFIGEfUHFS73zkFAGxQdOoDCDGgnTSwSc=
> =jp4S
> -----END PGP SIGNATURE-----
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130916/515e901d/attachment.htm>