<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi,</div><div><br></div><div>Generally, when a module is critical, and a lot solicited, I create a pg2 "pool" of supervised gen_server that will join the group, and get the pid using get_closest_pid in order to have multiple process.</div><div><br></div><div>Further more, The server has 16GB of RAM, and when the server is starting to get crazy, it only has 1,5 GB tops of RAM used. it really needs to go crazy a long time before using swap I guess, but I don't see it until another node cluster is telling me that the freeze node is timeout.</div><div><br></div><div>However, since we are at it… I may have find something really weird looking at my crash dump.</div><div>I'm using the emysql application.</div><div>My initialization of the emysql application is pretty basic : </div><div><br></div><div><div>application:start(emysql),</div><div>emysql:add_pool(my_db,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> 30,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> "login",</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> "password",</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> "<a href="http://my.db-host.com">my.db-host.com</a>",</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> 3306,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> "table",</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> latin1)</div></div><div><br></div><div>Has you can see, I only have 30 connections asked in the pool. However in the crash dump here's what I have found in the fun table : </div><div><br></div><div>Module<span class="Apple-tab-span" style="white-space:pre"> </span>Uniq<span class="Apple-tab-span" style="white-space:pre"> </span>Index<span class="Apple-tab-span" style="white-space:pre"> </span>Address<span class="Apple-tab-span" style="white-space:pre"> </span>Native_address<span class="Apple-tab-span" style="white-space:pre"> </span>Refc</div><div><br></div><div><div><div><div><div><div><div><font face="Times">emysql_util<span class="Apple-tab-span" style="white-space:pre"> </span>8432855<span class="Apple-tab-span" style="white-space:pre"> </span>1<span class="Apple-tab-span" style="white-space:pre"> </span>0x00007f1d4f9f6f00<span class="Apple-tab-span" style="white-space:pre"> </span> <span class="Apple-tab-span" style="white-space:pre"> </span>3476</font></div><div><font face="Times">emysql_util<span class="Apple-tab-span" style="white-space:pre"> </span>8432855<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>0<span class="Apple-tab-span" style="white-space:pre"> </span>0x00007f1d4f9f7218<span class="Apple-tab-span" style="white-space:pre"> </span> <span class="Apple-tab-span" style="white-space:pre"> </span>3476</font></div><div><font face="Times">emysql_util<span class="Apple-tab-span" style="white-space:pre"> </span>8432855<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>3<span class="Apple-tab-span" style="white-space:pre"> </span>0x00007f1d4f9f6e48<span class="Apple-tab-span" style="white-space:pre"> </span> <span class="Apple-tab-span" style="white-space:pre"> </span>2</font></div><div><font face="Times">emysql_util<span class="Apple-tab-span" style="white-space:pre"> </span>8432855<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>2<span class="Apple-tab-span" style="white-space:pre"> </span>0x00007f1d4f9f6ea8<span class="Apple-tab-span" style="white-space:pre"> </span> <span class="Apple-tab-span" style="white-space:pre"> </span>1</font></div><div><font face="Times">emysql<span class="Apple-tab-span" style="white-space:pre"> </span>79898780<span class="Apple-tab-span" style="white-space:pre"> </span>0<span class="Apple-tab-span" style="white-space:pre"> </span>0x00007f1d4f9b56f8<span class="Apple-tab-span" style="white-space:pre"> </span> <span class="Apple-tab-span" style="white-space:pre"> </span>841</font></div></div></div></div></div></div></div><div><br></div><div>Is that something normal to have with only 30 connections in one pool ?</div><div><br></div><div>Thank you all.</div><div><br></div><br><div><div>Le 23 mai 2013 à 04:21, Bob Ippolito <<a href="mailto:bob@redivi.com">bob@redivi.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">This kind of thing tends to happen when you continuously send messages to a process faster than it can handle them. The most common case that I've seen this is where you have a lot of processes communicating with a single gen_server process. If your server has swap enabled, this may appear to make the node "freeze completely but not crash".<div>
<br></div><div style="">In the past I've diagnosed this by monitoring the message_queue_len of registered processes, but I'm sure there are tools that can help do this for you.</div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Wed, May 22, 2013 at 7:00 PM, Morgan Segalis <span dir="ltr"><<a href="mailto:msegalis@gmail.com" target="_blank">msegalis@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hello everyone,<div><br></div><div>I'm having a bit of an issue with my production servers.</div><div><br></div><div>At some point, it seems to enter into an infinite loop that I can't find, or reproduce by myself on the tests servers.</div>
<div><br></div><div>The bug appear completely random, 1 hour, or 10 hour after restarting the Erlang node.</div><div>The loop will eat up all my server's memory in no time, and freeze completely the Erlang node without crashing it. (most of the time)</div>
<div><br></div><div>One time I got an crash dump, and tried to investigate it with cdv, but I didn't get much informations about which process or module was eating up all the memory.</div><div>I just know that it crashed because of the crash message : "<span style="font-family:Times">eheap_alloc: Cannot allocate 6801972448 bytes of memory (of type "heap")."</span></div>
<div><br></div><div>I'm surely too new to Erlang to investigate something like this with cdv, I really would like some pointers on how I can understand this problem and fix it asap.</div><div><br></div><div>If you need any informations about the crash dump, let me know what you need, I'll copy/paste…</div>
<div><br></div><div>I'm using <span style="font-family:Times">Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:true]</span></div><div><br></div><div>Thank you all for your help !</div>
<div><br></div></div><br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></body></html>