<div dir="ltr">Hi James,<div><br></div><div>I'm seeing similar issues on a cluster installation using R15B01. Did you get anywhere and figure out the issue?</div><div><br></div><div>Cheers,</div><div>Tino</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Aug 3, 2011 at 8:35 PM, James Aimonetti <span dir="ltr"><<a href="mailto:james@2600hz.com" target="_blank">james@2600hz.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
</div>So I started a VM with the +Mim true flag and wondered if folks can help<br>
narrow down where I should be looking for possible offenders?<br>
<br>
Using this in the shell:<br>
<br>
f(), {_,Mem}= instrument:descr(instrument:memory_data()), Sorted =<br>
lists:reverse(lists:keysort(3,Mem)).<br>
[{heap,140578906112088,41084720,<<a href="tel:0.5051.56" value="+49505156">0.5051.56</a>>},<br>
{old_heap,140578980069464,32867800,<<a href="tel:0.5051.56" value="+49505156">0.5051.56</a>>},<br>
{old_heap,140579035463768,10770152,<<a href="tel:0.8797.57" value="+49879757">0.8797.57</a>>},<br>
{heap,140578978496600,1571344,<<a href="tel:0.8797.57" value="+49879757">0.8797.57</a>>},<br>
{heap,140579077124184,971144,<0.25.0>},<br>
{old_heap,140579095326808,971144,<0.455.0>},<br>
{old_heap,140579078697048,971144,<0.25.0>},<br>
{old_heap,140579090284632,600200,<0.404.0>},<br>
{timer_wheel,140579109355368,524288,undefined},<br>
{port_tab,140579107569760,466944,undefined},<br>
{code,140579086084496,326762,<0.25.0>},<br>
{scheduler_data,140579110159688,287680,undefined},<br>
{proc_tab,140579109892896,262144,undefined},<br>
{code,140579070881888,251787,<0.25.0>},<br>
{code,140579106199176,239377,<0.0.0>},<br>
{old_heap,140579074121920,229256,<0.396.0>},<br>
{code,140579082245832,193718,<0.25.0>},<br>
{code,140579082439600,187723,<0.25.0>},<br>
{code,140579075601296,183984,<0.25.0>},<br>
{code,140579085875512,183888,<0.25.0>},<br>
|...]<br>
<br>
The first four lines are the shell and a process spawned by the shell.<br>
<br>
If I run:<br>
<br>
lists:foldl(fun({_,_,M,_},Acc) -> Acc + M end, 0, Sorted).<br>
112277446<br>
<br>
Is that number valid? Should I be looking at lines with type heap or<br>
old_heap only?<br>
<br>
The memory output from /proc/PID/status:<br>
<br>
VmPeak: 728864 kB<br>
VmSize: 335188 kB<br>
VmLck: 0 kB<br>
VmHWM: 421760 kB<br>
VmRSS: 250352 kB<br>
VmData: 306400 kB<br>
VmStk: 136 kB<br>
VmExe: 1800 kB<br>
VmLib: 3748 kB<br>
VmPTE: 680 kB<br>
VmSwap: 1520 kB<br>
<br>
Any direction for where to continue searching?<br>
<br>
Thanks for the help so far,<br>
<br>
James<br>
<div class="im"><br>
On 08/01/2011 10:32 AM, Mihai Balea wrote:<br>
> That looks pretty benign. I suspect the OS reported memory usage for the beam process is vastly higher, correct?<br>
> Do you run any native code? NIFs, linked-in drivers? Maybe there's a leak in there.<br>
><br>
> On Aug 1, 2011, at 11:55 AM, James Aimonetti wrote:<br>
><br>
</div><div><div class="h5">> No, no outliers.<br>
><br>
> 1> erlang:memory().<br>
> [{total,29393336},<br>
> {processes,18646408},<br>
> {processes_used,18560824},<br>
> {system,10746928},<br>
> {atom,658865},<br>
> {atom_used,652857},<br>
> {binary,1059104},<br>
> {code,6608997},<br>
> {ets,434848}]<br>
><br>
> Unless I'm reading that output incorrectly?<br>
><br>
> On 08/01/2011 01:39 AM, Attila Rajmund Nohl wrote:<br>
>>>> What does erlang:memory() show? Binaries, processes or ets uses that<br>
>>>> much memory?<br>
>>>><br>
>>>> 2011/8/1, James Aimonetti <<a href="mailto:james@2600hz.com">james@2600hz.com</a>>:<br>
>>>> List,<br>
>>>><br>
>>>> I'm at my wits end (they're short, no doubt, but still). I have a VM<br>
>>>> running with several OTP applications that we've written. We see, over<br>
>>>> the course of the day, memory consumption go up until its taking ~30%<br>
>>>> system memory and 300-500MB of swap. Iterating over all processes and<br>
>>>> checking their total_heap_size, no single process was greater than ~1<br>
>>>> MB, with ~300 processes runnning. Forcing garbage collection on all<br>
>>>> processes released maybe 15MB of system memory.<br>
>>>><br>
>>>> erlang:memory() lists total memory around 18-20MB; ets tables were all<br>
>>>> minimal, no dets tables. No processes has a message_queue_len > 0. No<br>
>>>> processes that we maintain use the process dictionary.<br>
>>>><br>
>>>> I started a second VM, migrating our OTP apps to the second VM, one at a<br>
>>>> time, hoping that stopping one would cause a release of lots of memory<br>
>>>> on the first VM. No such luck; it released a couple more MBs but still a<br>
>>>> large presence and no change in swap usage. I then stopped everything I<br>
>>>> could until only stdlib and kernel were running on the first VM, but to<br>
>>>> no avail.<br>
>>>><br>
>>>> Each application has 1-3 long running gen_servers that listen on an AMQP<br>
>>>> queue and spawn worker processes to handle messages. They carry almost<br>
>>>> no state. Some of the handlers will read binary blobs, open a socket,<br>
>>>> wait for a connection, then transmit the blobs, closing the socket and<br>
>>>> dying afterwards (they timeout after 5 minutes if no connection is<br>
>>>> attempted, and are in a supervision tree so I know there aren't a mass<br>
>>>> of them lying around). The blobs are between 20KB and 2MB. The rest of<br>
>>>> the workers are similarly ephemeral.<br>
>>>><br>
>>>> What other options are there to find where memory has been allocated? We<br>
>>>> use binaries almost exclusively for strings, don't construct large<br>
>>>> lists. I've poured over the efficiency guide looking for ideas of where<br>
>>>> we've gone astray but am coming up blank so far.<br>
>>>><br>
>>>> The system is idle; the only processes should be (and are, as far as I<br>
>>>> can tell) the gen_servers meant to run for a long time.<br>
>>>><br>
>>>> Erlang R14B01 (erts-5.8.2) [source] [64-bit] [smp:4:4] [rq:4]<br>
>>>> [async-threads:8] [kernel-poll:true]<br>
>>>><br>
>>>> Any ideas are welcome,<br>
>>>><br>
>>>> James<br>
>>>><br>
> _______________________________________________<br>
> erlang-questions mailing list<br>
> <a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
> <a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
>>>>><br>
>>>> _______________________________________________<br>
>>>> erlang-questions mailing list<br>
>>>> <a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
>>>> <a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
><br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
- --<br>
James Aimonetti<br>
Distributed Systems Engineer / DJ MC_<br>
<br>
2600hz | <a href="http://2600hz.com" target="_blank">http://2600hz.com</a><br>
<a href="mailto:sip%3Ajames@2600hz.com">sip:james@2600hz.com</a><br>
tel: 415.886.7905<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.10 (GNU/Linux)<br>
Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
<br>
</div></div>iQEcBAEBAgAGBQJOOZTzAAoJENc77s1OYoGgNSQH+QGcD5WDBSJp+z6RCjAqcRS+<br>
TM0fxxLolnWN9wdeje/v91ZA+G+usz/LbR0LhctwD7gJk29Ni56CO2cNP1osw5/c<br>
VWMSvxllDqn6ZmqAGl+h0lpYH9ZLWiZNZ1f4BjihHLjyglhdAKdRYfbQSzZsilQA<br>
Zu2AcfBvSQ564ByOEnU4ZEqrVaY9+EnI7WOjfd9HMHG2GBPgpDJauUGWfF21Qglf<br>
w7FESB/Gv6TMUHgdBEpBXmi1A1XgJFlQh8kK0xz6rNEp7GbHHrzk9Nl/644HzwGJ<br>
jsMpmBf6r+NXDwlR4Brd7eIdoIcnJ7DFIGEfUHFS73zkFAGxQdOoDCDGgnTSwSc=<br>
=jp4S<br>
<div class="HOEnZb"><div class="h5">-----END PGP SIGNATURE-----<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></div></blockquote></div><br></div>