[erlang-questions] Fwd: gen_server doesn't respond
Park, Sungjin
jinni.park@REDACTED
Tue Jun 11 03:40:06 CEST 2013
I succeeded in reproducing the problem throughout three days expedition and
about to share the experience now.
All the gen_server:call/4 in the stacktrace are found to be calls to lager
actually. The lager behaves asynchronously at first but goes synchronous
at some limit. Once it goes synchronous there are hundreds of thousands of
process waiting for reply. It makes the system hang. Actually the vm and
shell is still responsive but the processes hang and any new process
attempting to leave a log hangs also. I was using lager with {async, 100}
and 1000 rolling files of 100MB each. I suppose shifting those files would
have taken quite some time for lager that it triggered lager's synchronous
mode. What made the problem more probable is that there was a cron
scheduler designed to copy the log files regularly to a different
directory, whose schedule could overlap with lager's log file shifting
schedule eventually to make things worse. I solved the problem by reducing
number of rolling files and changing the cron scheduler to move files
rather than to copy and to run more often to prevent burst.
One thing I hope there be an option for lager not to go synchronous but
just drop logs at some limit. For my kind of system, synchronous logging
is simply unusable.
On Thu, Jun 6, 2013 at 5:48 AM, Robert Virding <
robert.virding@REDACTED> wrote:
> Funny discussion this. In this case I think Dmitry hit the nail on the
> head when he said that
>
> {current_function,{*gen,do_call,4*}}
>
> exactly describes what your problem is. The gen_server is suspended
> waiting in a gen_server:call to another server which is not replying.
> (gen_server:call -> gen:call/4 -> gen:do_call/4) With a little basic
> tracing (sys:trace) you should be able to work out where/why the server is
> calling another server and why it is hanging.
>
> Robert
>
> ------------------------------
>
> *From: *"Dmitry Kolesnikov" <dmkolesnikov@REDACTED>
> *To: *"Sungjin Park" <jinni.park@REDACTED>
> *Cc: *"erlang-questions" <erlang-questions@REDACTED>
> *Sent: *Tuesday, 4 June, 2013 10:45:40 AM
>
> *Subject: *Re: [erlang-questions] gen_server doesn't respond
>
> Hello,
>
> {current_function,{*gen,do_call,4*}}
>
> Your process made a call to another process with infinite timeout.
>
> The process do not respond because you got a deadlock.
> You have to check:
> * no circular dependencies on process call
> * the parent supervisor is not called from process init
> * no infinite gen_server:call is used at your api.
>
> - Dmitry
>
> On Jun 4, 2013, at 4:54 AM, "Park, Sungjin" <jinni.park@REDACTED> wrote:
>
> Shell input/output logs below.
>
> ============================
> 4> process_info(pid(0,453,4757)).
> [{current_function,{*gen,do_call,4*}},
> {initial_call,{proc_lib,init_p,5}},
> {status,*waiting*},
> {message_queue_len,1},
> {messages,[*{'$gen_call',{<0.26172.4841>,*
> * #Ref<0.0.2946.93178>},*
> * state}*]},
> {links,[<0.25337.4839>]},
> {dictionary,[{'$ancestors',[<0.452.4757>,<0.424.0>,
> <0.423.0>,ranch_sup,<0.420.0>]},
> {'$initial_call',{my_session,init,1}}]},
> {trap_exit,true},
> {error_handler,error_handler},
> {priority,normal},
> {group_leader,<0.419.0>},
> {total_heap_size,1364},
> {heap_size,987},
> {stack_size,32},
> {reductions,1487},
> {garbage_collection,[{min_bin_vheap_size,46368},
> {min_heap_size,233},
> {fullsweep_after,65535},
> {minor_gcs,8}]},
> {suspending,[]}]
> =============================
>
> As you can see the process has a message {..., state} in it's message
> queue but still in waiting state. It doesn't respond to
> gen_server:call(Pid, state) from this point on. The process_info shows
> that the process is now in gen:do_call/4 which is not under my control. The
> module has all the fallback handlers for handle_call/3, handle_cast/2,
> handle_info/2 etc, so that every message be matched. There are a couple of
> places in my project where gen_server:call/2 used against this process. I
> suppose that the calls should timeout in 5 sec by default as is stated in
> the gen_server specification. So there should be no such case that the
> process stuck indefinitely.
>
>
>
> On Mon, Jun 3, 2013 at 7:39 PM, Aaron France <aaron.l.france@REDACTED>wrote:
>
>> Does it make any synchronous calls to other gen_servers?
>>
>>
>> On Mon, Jun 3, 2013 at 12:34 PM, Tim Watson <watson.timothy@REDACTED>wrote:
>>
>>> Does the callback module execute any selective receives inside handlers
>>> perhaps? If not, are you sure the message has actually arrived in that
>>> process' mailbox, can you see unhandled messages in the message queue
>>> (I.e., length > 0 whilst waiting)?
>>>
>>> On 3 Jun 2013, at 10:34, "Park, Sungjin" <jinni.park@REDACTED> wrote:
>>>
>>> > Guys,
>>> >
>>> > I'm encountering cases when gen_server doesn't respond.
>>> > In such cases, erlang:process_info/1 shows the process state 'waiting'
>>> with some messages in the message_queue. The messages have not been
>>> consumed for tens of minutes at least. Anybody has any idean about this?
>>> >
>>> >
>>> > --
>>> > Park, Sungjin
>>> >
>>> -------------------------------------------------------------------------------------------------------------------
>>> > Peculiar travel suggestions are dancing lessons from god.
>>> > -- The Books of Bokonon
>>> >
>>> -------------------------------------------------------------------------------------------------------------------
>>> > _______________________________________________
>>> > erlang-questions mailing list
>>> > erlang-questions@REDACTED
>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>
>
> --
> Park, Sungjin
>
> -------------------------------------------------------------------------------------------------------------------
> Peculiar travel suggestions are dancing lessons from god.
> -- The Books of Bokonon
>
> -------------------------------------------------------------------------------------------------------------------
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
--
Park, Sungjin
-------------------------------------------------------------------------------------------------------------------
Peculiar travel suggestions are dancing lessons from god.
-- The Books of Bokonon
-------------------------------------------------------------------------------------------------------------------
--
Park, Sungjin
-------------------------------------------------------------------------------------------------------------------
Peculiar travel suggestions are dancing lessons from god.
-- The Books of Bokonon
-------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130611/a5916c56/attachment.htm>
More information about the erlang-questions
mailing list