<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>No problems - thanks for letting me know Siri!</div><div><br></div><div>Cheers,</div><div>Tim</div><br><div><div>On 19 Jun 2013, at 10:47, Siri Hansen wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hi Tim - I'm so sorry for the long delay here. I agree that this is a bug in the application master and I have written a ticket for it so it will be prioritized into out backlog.<div style="">Thanks for the report!</div>
<div style="">Regards</div><div style="">/siri</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/5/29 Tim Watson <span dir="ltr"><<a href="mailto:watson.timothy@gmail.com" target="_blank">watson.timothy@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Any word from the OTP folks on this one?<br>
<div class="HOEnZb"><div class="h5"><br>
On 24 May 2013, at 18:41, Tim Watson <<a href="mailto:watson.timothy@gmail.com">watson.timothy@gmail.com</a>> wrote:<br>
<br>
> Gah, sorry folks - this has nothing to do with release handling, that was a red herring. Someone just pointed out that the call to get_child originates in a status check in our code.<br>
><br>
> This still looks like a bug to me though, since if you're going to handle "other" messages in terminate_loop you ought to ensure they can't deadlock the vm's shutdown sequence.<br>
><br>
> Cheers,<br>
> Tim<br>
><br>
> On 24 May 2013, at 15:45, Fred Hebert <<a href="mailto:mononcqc@ferd.ca">mononcqc@ferd.ca</a>> wrote:<br>
><br>
>> Quick question: are you running a release?<br>
>><br>
>> If so, last time I've seen deadlocks like that was solved by making sure<br>
>> *all* my applications did depend on stdlib and kernel in their app file.<br>
>> When I skipped them, sometimes I'd find that things would lock up.<br>
>><br>
>> My guess was that dependencies from stdlib or kernel got unloaded before<br>
>> my app and broke something, but I'm not sure -- In my case, I wasn't<br>
>> able to inspect the node as it appeared to be 100% blocked.<br>
>><br>
>> Adding the apps ended up fixing the problem on the next shutdown. I'm<br>
>> not sure if it might be a good fix for you, but it's a stab in the dark,<br>
>><br>
>> Regards,<br>
>> Fred.<br>
>><br>
>> On 05/24, Tim Watson wrote:<br>
>>> We came across this at a customer's site, where one of the nodes was apparently in the process of stopping and had been in that state for at least 24 hours. The short version is that an application_master appears to be stuck waiting for a child pid (is that the X process, or the root supervisor?) which is *not* linked to it...<br>
>>><br>
>>> The application controller is in the process of stopping an application, during which process a `get_child' message appears to have come in to that application's application_master from somewhere - we are *not* running appmon, so I'm really confused how this can happen, as the only other place where I see (indirect) calls are via the sasl release_handler!? At the bottom of this email is a dump for the application_controller and the application_master for the app it is trying to shut down. I can verify that the pid which the application_master is waiting on is definitely not linked to it - i.e., process_info(links, AppMasterPid) doesn't contain the process <0.256.0> that the master appears to be waiting on.<br>
>>><br>
>>> My reading of the code is that the application_master cannot end up in get_child_i unless a get_child request was made which arrives whilst it is in its terminate loop. As I said, we're not using appmon, therefore I assume this originated in the sasl application's release_handler_1, though I'm not sure quite which route would take us there. The relevant bit of code in application_master appears to be:<br>
>>><br>
>>> get_child_i(Child) -><br>
>>> Child ! {self(), get_child},<br>
>>> receive<br>
>>> {Child, GrandChild, Mod} -> {GrandChild, Mod}<br>
>>> end.<br>
>>><br>
>>> This in turn originates, I'd guess, in the third receive clause of terminate_loop/2. Anyway, should that code not be dealing with a potentially dead pid for Child, either by handling links effectively - perhaps there is an EXIT signal in the mailbox already which is being ignored here in get_child_i/1 - or by some other means?<br>
>>><br>
>>> What follows below is the trace/dump output. Feel free to poke me for more info as needed.<br>
>>><br>
>>> Cheers,<br>
>>> Tim<br>
>>><br>
>>> [TRACE/DUMP]<br>
>>><br>
>>> pid: <6676.7.0><br>
>>> registered name: application_controller<br>
>>> stacktrace: [{application_master,call,2,<br>
>>> [{file,"application_master.erl"},{line,75}]},<br>
>>> {application_controller,stop_appl,3,<br>
>>> [{file,"application_controller.erl"},<br>
>>> {line,1393}]},<br>
>>> {application_controller,handle_call,3,<br>
>>> [{file,"application_controller.erl"},<br>
>>> {line,810}]},<br>
>>> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]}]<br>
>>> -------------------------<br>
>>> Program counter: 0x00007f9bf9a53720 (application_master:call/2 + 288)<br>
>>> CP: 0x0000000000000000 (invalid)<br>
>>> arity = 0<br>
>>><br>
>>> 0x00007f9bd7948360 Return addr 0x00007f9bfb97de40 (application_controller:stop_appl/3 + 176)<br>
>>> y(0) #Ref<0.0.20562.258360><br>
>>> y(1) #Ref<0.0.20562.258361><br>
>>> y(2) []<br>
>>><br>
>>> 0x00007f9bd7948380 Return addr 0x00007f9bfb973c68 (application_controller:handle_call/3 + 1392)<br>
>>> y(0) temporary<br>
>>> y(1) rabbitmq_web_dispatch<br>
>>><br>
>>> 0x00007f9bd7948398 Return addr 0x00007f9bf9a600c8 (gen_server:handle_msg/5 + 272)<br>
>>> y(0) {state,[],[],[],[{ssl,<0.507.0>},{public_key,undefined},{crypto,<0.501.0>},{rabbitmq_web_dispatch,<0.255.0>},{webmachine,<0.250.0>},{mochiweb,undefined},{xmerl,undefined},{inets,<0.237.0>},{amqp_client,<0.233.0>},{mnesia,<0.60.0>},{sasl,<0.34.0>},{stdlib,undefined},{kernel,<0.9.0>}],[],[{ssl,temporary},{public_key,temporary},{crypto,temporary},{rabbitmq_web_dispatch,temporary},{webmachine,temporary},{mochiweb,temporary},{xmerl,temporary},{inets,temporary},{amqp_client,temporary},{mnesia,temporary},{sasl,permanent},{stdlib,permanent},{kernel,permanent}],[],[{rabbit,[{ssl_listeners,[5671]},{ssl_options,[{cacertfile,"/etc/rabbitmq/server.cacrt"},{certfile,"/etc/rabbitmq/server.crt"},{keyfile,"/etc/rabbitmq/server.key"},{verify,verify_none},{fail_if_no_peer_cert,false}]},{default_user,<<2 bytes>>},{default_pass,<<8 bytes>>},{vm_memory_high_watermark,5.000000e-01}]},{rabbitmq_management,[{listener,[{port,15672},{ssl,true}]}]}]}<br>
>>> y(1) rabbitmq_web_dispatch<br>
>>> y(2) [{ssl,temporary},{public_key,temporary},{crypto,temporary},{rabbitmq_web_dispatch,temporary},{webmachine,temporary},{mochiweb,temporary},{xmerl,temporary},{inets,temporary},{amqp_client,temporary},{mnesia,temporary},{sasl,permanent},{stdlib,permanent},{kernel,permanent}]<br>
>>> y(3) [{ssl,<0.507.0>},{public_key,undefined},{crypto,<0.501.0>},{rabbitmq_web_dispatch,<0.255.0>},{webmachine,<0.250.0>},{mochiweb,undefined},{xmerl,undefined},{inets,<0.237.0>},{amqp_client,<0.233.0>},{mnesia,<0.60.0>},{sasl,<0.34.0>},{stdlib,undefined},{kernel,<0.9.0>}]<br>
>>><br>
>>> 0x00007f9bd79483c0 Return addr 0x00000000008827d8 (<terminate process normally>)<br>
>>> y(0) application_controller<br>
>>> y(1) {state,[],[],[],[{ssl,<0.507.0>},{public_key,undefined},{crypto,<0.501.0>},{rabbitmq_web_dispatch,<0.255.0>},{webmachine,<0.250.0>},{mochiweb,undefined},{xmerl,undefined},{inets,<0.237.0>},{amqp_client,<0.233.0>},{mnesia,<0.60.0>},{sasl,<0.34.0>},{stdlib,undefined},{kernel,<0.9.0>}],[],[{ssl,temporary},{public_key,temporary},{crypto,temporary},{rabbitmq_web_dispatch,temporary},{webmachine,temporary},{mochiweb,temporary},{xmerl,temporary},{inets,temporary},{amqp_client,temporary},{mnesia,temporary},{sasl,permanent},{stdlib,permanent},{kernel,permanent}],[],[{rabbit,[{ssl_listeners,[5671]},{ssl_options,[{cacertfile,"/etc/rabbitmq/server.cacrt"},{certfile,"/etc/rabbitmq/server.crt"},{keyfile,"/etc/rabbitmq/server.key"},{verify,verify_none},{fail_if_no_peer_cert,false}]},{default_user,<<2 bytes>>},{default_pass,<<8 bytes>>},{vm_memory_high_watermark,5.000000e-01}]},{rabbitmq_management,[{listener,[{port,15672},{ssl,true}]}]}]}<br>
>>> y(2) application_controller<br>
>>> y(3) <0.2.0><br>
>>> y(4) {stop_application,rabbitmq_web_dispatch}<br>
>>> y(5) {<0.5864.275>,#Ref<0.0.20562.258345>}<br>
>>> y(6) Catch 0x00007f9bf9a600c8 (gen_server:handle_msg/5 + 272)<br>
>>> -------------------------<br>
>>><br>
>>> pid: <6676.255.0><br>
>>> registered name: none<br>
>>> stacktrace: [{application_master,get_child_i,1,<br>
>>> [{file,"application_master.erl"},{line,392}]},<br>
>>> {application_master,handle_msg,2,<br>
>>> [{file,"application_master.erl"},{line,216}]},<br>
>>> {application_master,terminate_loop,2,<br>
>>> [{file,"application_master.erl"},{line,206}]},<br>
>>> {application_master,terminate,2,<br>
>>> [{file,"application_master.erl"},{line,227}]},<br>
>>> {application_master,handle_msg,2,<br>
>>> [{file,"application_master.erl"},{line,219}]},<br>
>>> {application_master,main_loop,2,<br>
>>> [{file,"application_master.erl"},{line,194}]},<br>
>>> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]<br>
>>> -------------------------<br>
>>> Program counter: 0x00007f9bf9a570e0 (application_master:get_child_i/1 + 120)<br>
>>> CP: 0x0000000000000000 (invalid)<br>
>>> arity = 0<br>
>>><br>
>>> 0x00007f9c1adc3dc8 Return addr 0x00007f9bf9a54eb0 (application_master:handle_msg/2 + 280)<br>
>>> y(0) <0.256.0><br>
>>><br>
>>> 0x00007f9c1adc3dd8 Return addr 0x00007f9bf9a54d20 (application_master:terminate_loop/2 + 520)<br>
>>> y(0) #Ref<0.0.20562.258362><br>
>>> y(1) <0.9596.275><br>
>>> y(2) {state,<0.256.0>,{appl_data,rabbitmq_web_dispatch,[],undefined,{rabbit_web_dispatch_app,[]},[rabbit_web_dispatch,rabbit_web_dispatch_app,rabbit_web_dispatch_registry,rabbit_web_dispatch_sup,rabbit_web_dispatch_util,rabbit_webmachine],[],infinity,infinity},[],0,<0.29.0>}<br>
>>><br>
>>> 0x00007f9c1adc3df8 Return addr 0x00007f9bf9a55108 (application_master:terminate/2 + 192)<br>
>>> y(0) <0.256.0><br>
>>><br>
>>> 0x00007f9c1adc3e08 Return addr 0x00007f9bf9a54f70 (application_master:handle_msg/2 + 472)<br>
>>> y(0) []<br>
>>> y(1) normal<br>
>>><br>
>>> 0x00007f9c1adc3e20 Return addr 0x00007f9bf9a54a60 (application_master:main_loop/2 + 1600)<br>
>>> y(0) <0.7.0><br>
>>> y(1) #Ref<0.0.20562.258360><br>
>>> y(2) Catch 0x00007f9bf9a54f70 (application_master:handle_msg/2 + 472)<br>
>>><br>
>>> 0x00007f9c1adc3e40 Return addr 0x00007f9bfb969420 (proc_lib:init_p_do_apply/3 + 56)<br>
>>> y(0) <0.7.0><br>
>>><br>
>>> 0x00007f9c1adc3e50 Return addr 0x00000000008827d8 (<terminate process normally>)<br>
>>> y(0) Catch 0x00007f9bfb969440 (proc_lib:init_p_do_apply/3 + 88)<br>
>>> -------------------------<br>
>>><br>
>><br>
>><br>
>><br>
>>> _______________________________________________<br>
>>> erlang-bugs mailing list<br>
>>> <a href="mailto:erlang-bugs@erlang.org">erlang-bugs@erlang.org</a><br>
>>> <a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a><br>
>><br>
_______________________________________________<br>
erlang-bugs mailing list<br>
<a href="mailto:erlang-bugs@erlang.org">erlang-bugs@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a><br>
</div></div></blockquote></div><br></div>
</blockquote></div><br></body></html>