Nothing worth to mention: net_kernel just in waiting state. In dump there are only 3 processes not in waiting state: user_drv runng, and 2 processes in scheduled state in erlang:apply (1 in unicode:ml_map/3 and other in cl_consumer:consumer_wait/0 called from OSERL)<div>
<br></div><div>Kind regards,</div><div>Kirill Zaborsky<br><div><br><div class="gmail_quote">2011/10/28 Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com">spawn.think@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I saw a similar behavior but with rex process (rpc server) having a very long queue. For your case i wonder what's loading the user_drv process, but as you mentioned the crash dump was truncated.<div>Do you see in the crash dump anything about net_kernel process? <div>
<div></div><div class="h5"><br>
<br><div class="gmail_quote">On Fri, Oct 28, 2011 at 1:41 PM, Kirill Zaborsky <span dir="ltr"><<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It's a server to collect information from GPS trackers (sent by SMS or HTTP/GPRS), uses Yaws to get information by HTTP. Processed information goes to PostgreSQL (epgsql is used). Then this info could be shown with qooxdoo/mapserver, webinterface backend is done with mochiweb.<div>
There is no custom NIFs of C nodes so I'm not sure how could this application hang Erlang emulator.</div><div>On other Erlang system done by our company we have experienced similar behaviour when user was connected to the server shell (using ndetool's attach command i.e. to_erl program) and then ssh session was broken, Long message queue for user_drv may be connected to something like that. The problem is that for the system I'm trying to get a diagnose there was no any shell/console connected. So It's unclear what could trigger such problem.</div>
<div><div></div><div>
<div><br></div><div>Kind regards,</div><div>Kirill Zaborsky<br><br><div class="gmail_quote">2011/10/28 Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com" target="_blank">spawn.think@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Maybe providing some information about what your application is doing might help?<div><div></div><div><br><br><div class="gmail_quote">On Fri, Oct 28, 2011 at 10:47 AM, Kirill Zaborsky <span dir="ltr"><<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">About message queue crash dump viewer says "The dump is truncated, no data available", so I've got no more infromation :-\<div>
epmd -names showed the node running but I could not contact it.</div><div><br></div>
<div>Kind regards,</div><div>Kirill Zaborsky<div><div></div><div><br><br><div class="gmail_quote">2011/10/28 Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com" target="_blank">spawn.think@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Are you able to expand message queue of user_drv process? That might give some information.<div>Did you check epmd status before dumping?<br><br><div class="gmail_quote"><div><div></div><div>On Fri, Oct 28, 2011 at 10:10 AM, Kirill Zaborsky <span dir="ltr"><<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div>Just 2 days passed and Erlang node got stuck once again.<div>This time I killed it with SIGUSR1 and received a crash dump.</div>
<div>Checking all the logs on host didn't bring any hints where the problem may be.</div><div>
And in crash dump the only suspicious thing is that user_drv has message queue length equal to 7550. The program counter points to user_drv:server_loop/5 + 48 - is there any way to get info what instruction in the source code it corresponds to?</div>
<div>BTW crash dump viewer says that crash dump was truncated is there any way to get full crash dump?</div><div>The system is running R14B03 if it matters.</div><div>Any advices are welcomed.</div><div><br></div><div>Kind regards,</div>
<div>Kirill Zaborsky </div><div><div></div><div><div><br><div class="gmail_quote">2011/10/26 Kirill Zaborsky <span dir="ltr"><<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Recently we have found some problems with our Erlang application:</div><div>For some time system works ok (e.g. before today it run with no problems for at least 17 days). Then something happens and it "stucks". It does not repond to pings, http interface (mochiweb) gives no replies. The only thing that can be observed is standard "ALIVE" message sent to stdout every 15 minutes when there is no output to stdout. Messages from logs show nothing special before logging stops.</div>
<div>The only thing I could do is just kill the emulator. That gives me opportunity to restart the system but gives no additional information about the roots of the problem.</div><div>On JVM it's possible to get program thread dump (using QUIT signal) is there some ways to "manually" force Erlang emulator to produce crash dump without using erlang:halt/1?</div>
<div>Are there some other ways to diagnose this problem which I should take a look at?</div><div><br></div><div>Kind regars,</div><div>Kirill Zaborksy</div>
</blockquote></div><br></div>
</div></div><br></div></div><div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></div></blockquote></div><div><br><br clear="all"><div><br></div>-- <br>Best Regards,<br>- Ahmed Omar<div><a href="http://nl.linkedin.com/in/adiaa" target="_blank">http://nl.linkedin.com/in/adiaa</a></div>
<div>Follow me on twitter</div>
<div><a href="http://twitter.com/#!/spawn_think" target="_blank">@spawn_think</a></div><br>
</div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Best Regards,<br>- Ahmed Omar<div><a href="http://nl.linkedin.com/in/adiaa" target="_blank">http://nl.linkedin.com/in/adiaa</a></div><div>Follow me on twitter</div>
<div><a href="http://twitter.com/#!/spawn_think" target="_blank">@spawn_think</a></div><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best Regards,<br>- Ahmed Omar<div><a href="http://nl.linkedin.com/in/adiaa" target="_blank">http://nl.linkedin.com/in/adiaa</a></div><div>Follow me on twitter</div>
<div><a href="http://twitter.com/#!/spawn_think" target="_blank">@spawn_think</a></div><br>
</div></div></div>
</blockquote></div><br></div></div>