<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Possible message leak? Check for unhandled messages, and log
them. See the section on unhandled messages <a
href="https://www.safaribooksonline.com/library/view/designing-for-scalability/9781449361556/ch04.html">here</a>.<br>
</p>
<br>
<div class="moz-cite-prefix">On 10/16/2016 03:05 AM, Paul Oliver
wrote:<br>
</div>
<blockquote
cite="mid:CABNk+6kagQxBe838JE2FrWTHifCkEKojt0soDG=9NwCo0c-vJw@mail.gmail.com"
type="cite">
<div dir="ltr">Hey Luca,
<div><br>
</div>
<div>Check out <a moz-do-not-send="true"
href="https://github.com/ferd/recon">https://github.com/ferd/recon</a> and <a
moz-do-not-send="true"
href="http://dieswaytoofast.blogspot.co.nz/2012/12/erlang-binaries-and-garbage-collection.html">http://dieswaytoofast.blogspot.com/2012/12/erlang-binaries-and-garbage-collection.html</a></div>
<div><br>
</div>
<div>Cheers,</div>
<div>Paul.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sun, Oct 16, 2016 at 8:53 PM Luca Spiller <<a
moz-do-not-send="true" href="mailto:luca@stackednotion.com">luca@stackednotion.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="gmail_msg">Hi everyone,
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">One of our nodes seems to have a
memory leak. After a couple of days the memory usage gets
so high that the OOM killer kills it, and it's restarted.
It seems to have been going on for a few years, as it
works fine the whole time so nobody noticed - it just uses
up all the memory on the box.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">A bit of background: the node is
making hundreds of HTTP requests per second. There are a
thousand or so worker processes responsible for this,
which make a request, inspect the response headers, and
based on these start other processes. The process then
sleeps for X time (seconds to minutes) and does the same
again. The response body can be any size, but we don't
care about that in the application (but I'd assume it gets
converted to a binary by lhttpc). I should also note that
some of the requests are made over TLS.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg"><a moz-do-not-send="true"
href="https://dl.dropboxusercontent.com/u/21557257/20161016-erl/observer-system.png"
class="gmail_msg" target="_blank">https://dl.dropboxusercontent.com/u/21557257/20161016-erl/observer-system.png</a><br
class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">This is the output from Observer, as
you can see it shows that binaries are using 2569 MB of
RAM. When the node has been restarted and running for a
few minutes this is usually < 10 MB. Most of the worker
processes (95%+) which make the requests are started
shortly after the node starts and hang around forever.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg"><a moz-do-not-send="true"
href="https://dl.dropboxusercontent.com/u/21557257/20161016-erl/observer-processes.png"
class="gmail_msg" target="_blank">https://dl.dropboxusercontent.com/u/21557257/20161016-erl/observer-processes.png</a><br
class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">This is the process list from
Observer, sorted by memory, it doesn't appear to show
anything interesting. The worker processes (XXX:init/1)
use roughly the same amount of memory after they've been
running for a few minutes.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">As I understand large binaries stick
around until the system is under 'high memory pressure'
before being GCed. In my case the node uses up half the
swap, and all the RAM - is that not high enough? After
that the OOM killer jumps in and deals with it forcibly.</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">So... what can I do to debug this?<br
class="gmail_msg">
</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">Thanks,</div>
<div class="gmail_msg"><br class="gmail_msg">
</div>
<div class="gmail_msg">Luca Spiller</div>
</div>
_______________________________________________<br
class="gmail_msg">
erlang-questions mailing list<br class="gmail_msg">
<a moz-do-not-send="true"
href="mailto:erlang-questions@erlang.org" class="gmail_msg"
target="_blank">erlang-questions@erlang.org</a><br
class="gmail_msg">
<a moz-do-not-send="true"
href="http://erlang.org/mailman/listinfo/erlang-questions"
rel="noreferrer" class="gmail_msg" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br
class="gmail_msg">
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</body>
</html>