<div dir="ltr">I succeeded in reproducing the problem throughout three days expedition and about to share the experience now.<br><div class="gmail_quote"><div dir="ltr"><div><br></div><div>All the gen_server:call/4 in the stacktrace are found to be calls to lager actually. The lager behaves asynchronously at first but goes synchronous at some limit. Once it goes synchronous there are hundreds of thousands of process waiting for reply. It makes the system hang. Actually the vm and shell is still responsive but the processes hang and any new process attempting to leave a log hangs also. I was using lager with {async, 100} and 1000 rolling files of 100MB each. I suppose shifting those files would have taken quite some time for lager that it triggered lager's synchronous mode. What made the problem more probable is that there was a cron scheduler designed to copy the log files regularly to a different directory, whose schedule could overlap with lager's log file shifting schedule eventually to make things worse. I solved the problem by reducing number of rolling files and changing the cron scheduler to move files rather than to copy and to run more often to prevent burst.</div>
<div><br></div><div>One thing I hope there be an option for lager not to go synchronous but just drop logs at some limit. For my kind of system, synchronous logging is simply unusable.</div><div><br></div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 6, 2013 at 5:48 AM, Robert Virding <span dir="ltr"><<a href="mailto:robert.virding@erlang-solutions.com" target="_blank">robert.virding@erlang-solutions.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><span style="overflow:visible"><span><span>Funny discussion this. In this case I think Dmitry </span></span></span>hit the nail on the head when he said that<br>
<br>{current_function,{<u><i>gen,do_call,4</i></u>}}<br><br>exactly describes what your problem is. The gen_server is suspended waiting in a gen_server:call to another server which is not replying. (gen_server:call -> gen:call/4 -> gen:do_call/4) With a little basic tracing (sys:trace) you should be able to work out where/why the server is calling another server and why it is hanging.<br>
<br>Robert<br><br><hr><blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left:2px solid #1010ff"><b>From: </b>"Dmitry Kolesnikov" <<a href="mailto:dmkolesnikov@gmail.com" target="_blank">dmkolesnikov@gmail.com</a>><br>
<b>To: </b>"Sungjin Park" <<a href="mailto:jinni.park@gmail.com" target="_blank">jinni.park@gmail.com</a>><br><b>Cc: </b>"erlang-questions" <<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
<b>Sent: </b>Tuesday, 4 June, 2013 10:45:40 AM<div><br><b>Subject: </b>Re: [erlang-questions] gen_server doesn't respond<br><br></div><div><div>Hello,<div><br></div><div><blockquote><div dir="ltr">
{current_function,{<u><i>gen,do_call,4</i></u>}}</div></blockquote></div><div>Your process made a call to another process with infinite timeout.</div><div><br></div><div>The process do not respond because you got a deadlock. </div>
<div>You have to check:</div><div> * no circular dependencies on process call</div><div> * the parent supervisor is not called from process init</div><div> * no infinite gen_server:call is used at your api. </div><div>
<br></div><div>- Dmitry</div><div><br><div><div>On Jun 4, 2013, at 4:54 AM, "Park, Sungjin" <<a href="mailto:jinni.park@gmail.com" target="_blank">jinni.park@gmail.com</a>> wrote:</div><br><blockquote><div dir="ltr">
Shell input/output logs below.<div><br></div><div>============================</div><div>4> process_info(pid(0,453,4757)).<br></div><div><div>[{current_function,{<u><i>gen,do_call,4</i></u>}},</div><div> {initial_call,{proc_lib,init_p,5}},</div>
<div> {status,<u><i>waiting</i></u>},</div><div> {message_queue_len,1},</div><div> {messages,[<u><i>{'$gen_call',{<0.26172.4841>,</i></u></div><div><u><i> #Ref<0.0.2946.93178>},</i></u></div>
<div><u><i> state}</i></u>]},</div><div> {links,[<0.25337.4839>]},</div><div> {dictionary,[{'$ancestors',[<0.452.4757>,<0.424.0>,</div><div> <0.423.0>,ranch_sup,<0.420.0>]},</div>
<div> {'$initial_call',{my_session,init,1}}]},</div><div> {trap_exit,true},</div><div> {error_handler,error_handler},</div><div> {priority,normal},</div><div> {group_leader,<0.419.0>},</div><div>
{total_heap_size,1364},</div><div> {heap_size,987},</div><div> {stack_size,32},</div><div> {reductions,1487},</div><div> {garbage_collection,[{min_bin_vheap_size,46368},</div><div> {min_heap_size,233},</div>
<div> {fullsweep_after,65535},</div><div> {minor_gcs,8}]},</div><div> {suspending,[]}]</div></div><div>=============================</div><div><br></div><div>As you can see the process has a message {..., state} in it's message queue but still in waiting state. It doesn't respond to gen_server:call(Pid, state) from this point on. The process_info shows that the process is now in gen:do_call/4 which is not under my control. The module has all the fallback handlers for handle_call/3, handle_cast/2, handle_info/2 etc, so that every message be matched. There are a couple of places in my project where gen_server:call/2 used against this process. I suppose that the calls should timeout in 5 sec by default as is stated in the gen_server specification. So there should be no such case that the process stuck indefinitely.</div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 3, 2013 at 7:39 PM, Aaron France <span dir="ltr"><<a href="mailto:aaron.l.france@gmail.com" target="_blank">aaron.l.france@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Does it make any synchronous calls to other gen_servers?</div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Jun 3, 2013 at 12:34 PM, Tim Watson <span dir="ltr"><<a href="mailto:watson.timothy@gmail.com" target="_blank">watson.timothy@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Does the callback module execute any selective receives inside handlers perhaps? If not, are you sure the message has actually arrived in that process' mailbox, can you see unhandled messages in the message queue (I.e., length > 0 whilst waiting)?<br>
<div><br>
On 3 Jun 2013, at 10:34, "Park, Sungjin" <<a href="mailto:jinni.park@gmail.com" target="_blank">jinni.park@gmail.com</a>> wrote:<br>
<br>
> Guys,<br>
><br>
> I'm encountering cases when gen_server doesn't respond.<br>
> In such cases, erlang:process_info/1 shows the process state 'waiting' with some messages in the message_queue. The messages have not been consumed for tens of minutes at least. Anybody has any idean about this?<br>
><br>
><br>
> --<br>
> Park, Sungjin<br>
> -------------------------------------------------------------------------------------------------------------------<br>
> Peculiar travel suggestions are dancing lessons from god.<br>
> -- The Books of Bokonon<br>
> -------------------------------------------------------------------------------------------------------------------<br>
</div>> _______________________________________________<br>
> erlang-questions mailing list<br>
> <a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
> <a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Park, Sungjin<div>-------------------------------------------------------------------------------------------------------------------</div><div>Peculiar travel suggestions are dancing lessons from god.</div>
<div> -- The Books of Bokonon</div><div>-------------------------------------------------------------------------------------------------------------------</div>
</div>
_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div><br></div><br>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></div></blockquote><br></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Park, Sungjin<div>-------------------------------------------------------------------------------------------------------------------</div>
<div>Peculiar travel suggestions are dancing lessons from god.</div><div> -- The Books of Bokonon</div><div>-------------------------------------------------------------------------------------------------------------------</div>
</div>
</div></div></div><br><br clear="all"><div><br></div>-- <br>Park, Sungjin<div>-------------------------------------------------------------------------------------------------------------------</div><div>Peculiar travel suggestions are dancing lessons from god.</div>
<div> -- The Books of Bokonon</div><div>-------------------------------------------------------------------------------------------------------------------</div>
</div>