[erlang-bugs] net_kernel waiting for response from self
Matthew Dempsky
matthew@REDACTED
Wed Sep 17 01:43:54 CEST 2008
Earlier we noticed one of our Erlang nodes was not connected to a
bunch of other nodes, and that issuing net_adm:ping(Node) to try to
reconnect it would just hang. It seems that net_kernel had issued a
gen_server:call to itself and was waiting indefinitely for a response.
Complete backtrace of Pid (net_kernel, <0.22.0>) is below. From the
back trace you can see net_kernel made a call to
gen_server:call(net_kernel, {connect, normal,
'mochisvn@REDACTED'}, infinity)
and sent the request with #Ref<0.0.622.220265>.
To confirm this, you can look at the process_info dump and see the
second element in the message queue is
{'$gen_call',{<0.22.0>,#Ref<0.0.622.220265>},
{connect,normal,'mochisvn@REDACTED'}}.
I forged a response by grabbing that references and calling
Pid ! {Ref, false},
which finally led net_kernel to proceed a little bit further, but it
immediately hung trying to handle
{'$gen_call',{<6341.14452.446>,#Ref<6341.0.542.13516>},
{is_auth,'discover@REDACTED'}}.
I repeated the process a few more times, forging responses to the
connect calls, but eventually gave up and just restarted the node
after determining there were 300,000 pending is_auth requests in the
message queue, and each needed a forged response to continue.
(mochiadsdb@REDACTED)88> bt(Pid).
Program counter: 0x00002aaaab841870 (gen:wait_resp_mon/3 + 64)
CP: 0x00002aaaab89dc30 (gen_server:call/3 + 160)
arity = 0
0x00002aaae3591870 Return addr 0x00002aaaab89dc30 (gen_server:call/3 + 160)
y(0) infinity
y(1) #Ref<0.0.622.220265>
y(2) 'mochiadsdb@REDACTED'
0x00002aaae3591890 Return addr 0x00002aaaab812d88 (erlang:dsend/2 + 136)
y(0) infinity
y(1) {connect,normal,'mochisvn@REDACTED'}
y(2) net_kernel
y(3) Catch 0x00002aaaab89dc30 (gen_server:call/3 + 160)
0x00002aaae35918b8 Return addr 0x00002aaaab89e160 (gen_server:reply/2 + 208)
y(0) {#Ref<6397.0.34.240428>,yes}
y(1) <6397.6917.55>
0x00002aaae35918d0 Return addr 0x00002aaaab8a2370
(gen_server:handle_msg/5 + 848)
y(0) Catch 0x00002aaaab89e160 (gen_server:reply/2 + 208)
0x00002aaae35918e0 Return addr 0x00002aaaab853310 (proc_lib:init_p/5 + 400)
y(0) net_kernel
y(1) []
y(2) net_kernel
y(3) <0.19.0>
y(4) []
y(5) []
y(6) {state,'mochiadsdb@REDACTED','mochiadsdb@REDACTED',longnames,{tick,<0.24.0>,15000},7000,sys_dist,[{<0.27218.334>,'mochibot@REDACTED'},{<0.5652.334>,'mochimonitor_ping@REDACTED'},{<0.30534.333>,'mochimond@REDACTED'},{<0.14584.332>,'mochibot@REDACTED'},{<0.10693.325>,'discover@REDACTED'},{<0.10679.325>,'mochimond@REDACTED'},{<0.8624.325>,'mochimond@REDACTED'},{<0.21639.321>,'discover@REDACTED'},{<0.21631.321>,'mochimond@REDACTED'},{<0.1479.310>,'mochimond@REDACTED'},{<0.1474.310>,'discover@REDACTED'},{<0.1794.308>,'mochimond@REDACTED'},{<0.14155.304>,'mochisvn@REDACTED'},{<0.8435.304>,'mochimond@REDACTED'},{<0.11959.295>,'mochimond@REDACTED'},{<0.11954.295>,'discover@REDACTED'},{<0.11952.295>,'mochisvn@REDACTED'},{<0.12655.294>,'mochimonitor_relay@REDACTED'},{<0.1597.285>,'mochicrypt@REDACTED'},{<0.31519.268>,'mochimond@REDACTED'},{<0.31492.268>,'mochiads_logger@REDACTED'},{<0.31418.268>,'mochimq2@REDACTED'},{<0.28344.266>,'mochimond@REDACTED'},{<0.28341.266>,'discover@REDACTED'},{<0.9573.266>,'mochisvn@REDACTED'},{<0.28064.264>,'mochimonitor_httpcheck@REDACTED'},{<0.24678.264>,'mochibot@REDACTED'},{<0.25804.263>,'mochimq2@REDACTED'},{<0.4231.260>,'mochiscore@REDACTED'},{<0.1352.260>,'admapper@REDACTED'},{<0.1348.260>,'mochiscore@REDACTED'},{<0.1346.260>,'mochiads_loggerv2@REDACTED'},{<0.32206.259>,'discover@REDACTED'},{<0.32169.259>,'discover@REDACTED'},{<0.20340.259>,'mochimond@REDACTED'},{<0.19993.259>,'mochimond@REDACTED'},{<0.16961.250>,'discover@REDACTED'},{<0.6399.250>,'discover@REDACTED'},{<0.6370.250>,'mochipass@REDACTED'},{<0.6286.250>,'discover@REDACTED'},{<0.6278.250>,'mochipass@REDACTED'},{<0.22656.248>,'mochimond@REDACTED'},{<0.22654.248>,'mochimond@REDACTED'},{<0.22653.248>,'mochimond@REDACTED'},{<0.22625.248>,'mochisvn@REDACTED'},{<0.22623.248>,'mochisvn@REDACTED'},{<0.22621.248>,'mochisvn@REDACTED'},{<0.22619.248>,'mochisvn@REDACTED'},{<0.22603.248>,'mochisvn@REDACTED'},{<0.22593.248>,'mochisvn@REDACTED'},{<0.22588.248>,'mochisvn@REDACTED'},{<0.22559.248>,'mochisvn@REDACTED'},{<0.22554.248>,'mochisvn@REDACTED'},{<0.22546.248>,'mochisvn@REDACTED'},{<0.22542.248>,'mochisvn@REDACTED'},{<0.22538.248>,'mochisvn@REDACTED'},{<0.22524.248>,'mochisvn@REDACTED'},{<0.22519.248>,'mochisvn@REDACTED'},{<0.22515.248>,'mochisvn@REDACTED'},{<0.22511.248>,'mochisvn@REDACTED'},{<0.22507.248>,'mochisvn@REDACTED'},{<0.22487.248>,'mochisvn@REDACTED'},{<0.22474.248>,'mochisvn@REDACTED'},{<0.22469.248>,'mochisvn@REDACTED'},{<0.19636.245>,'discover@REDACTED'},{<0.18890.245>,'mochimond@REDACTED'},{<0.18885.245>,'mochimond@REDACTED'},{<0.18856.245>,'discover@REDACTED'},{<0.18850.245>,'mochiads@REDACTED'},{<0.18830.245>,'discover@REDACTED'},{<0.18824.245>,'mochiads@REDACTED'},{<0.18818.245>,'discover@REDACTED'},{<0.18814.245>,'mochiads@REDACTED'},{<0.18800.245>,'discover@REDACTED'},{<0.18791.245>,'mochimond@REDACTED'},{<0.18789.245>,'mochiads@REDACTED'},{<0.18786.245>,'mochiads@REDACTED'},{<0.18721.245>,'mochimond@REDACTED'},{<0.18711.245>,'mochiads_logger@REDACTED'},{<0.18575.245>,'mochiads_logger@REDACTED'},{<0.18556.245>,'mochimonitor_alert@REDACTED'},{<0.18535.245>,'mochimond@REDACTED'},{<0.18423.245>,'mochiads_logger@REDACTED'},{<0.12300.159>,'mochimond@REDACTED'},{<0.23778.120>,'mochimond@REDACTED'},{<0.3295.120>,'juanita@REDACTED'},{<0.23280.101>,'mochiscore@REDACTED'},{<0.23276.101>,'mochiscore@REDACTED'},{<0.3402.67>,'discover@REDACTED'},{<0.3398.67>,'mochimond@REDACTED'},{<0.3339.67>,'discover@REDACTED'},{<0.3335.67>,'mochimond@REDACTED'},{<0.3328.67>,'discover@REDACTED'},{<0.8774.2>,'mochiads@REDACTED'},{<0.673.0>,'mochimonitor_alert@REDACTED'},{<0.679.0>,'mochimonitor_dbweb@REDACTED'},{<0.612.0>,'mochipass@REDACTED'},{<0.350.0>,'discover@REDACTED'},{<0.342.0>,'discover@REDACTED'},{<0.344.0>,'mochimond@REDACTED'},{<0.333.0>,'mochimond@REDACTED'},{<0.288.0>,'discover@REDACTED'},{<0.227.0>,'discover@REDACTED'},{<0.77.0>,'discover@REDACTED'},{<0.183.0>,'discover@REDACTED'},{<0.117.0>,'mochimond@REDACTED'},{<0.103.0>,'discover@REDACTED'},{<0.98.0>,'discover@REDACTED'},{<0.91.0>,'mochimond@REDACTED'},{<0.84.0>,'mochimond@REDACTED'},{<0.75.0>,'mochimond@REDACTED'},{<0.68.0>,'discover@REDACTED'}],[],[{listen,#Port<0.7>,<0.23.0>,{net_address,{{0,0,0,0},39758},"reaver",tcp,inet},inet_tcp_dist}],[],0,all}
0x00002aaae3591920 Return addr 0x00000000008572f8 (<terminate process normally>)
y(0) Catch 0x00002aaaab853330 (proc_lib:init_p/5 + 432)
y(1) gen
y(2) init_it
y(3) [gen_server,<0.19.0>,<0.19.0>,{local,net_kernel},net_kernel,{'mochiadsdb@REDACTED',longnames,15000},[]]
ok
(mochiadsdb@REDACTED)89> erlang:process_info(Pid).
[{registered_name,net_kernel},
{current_function,{gen,wait_resp_mon,3}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,458946},
{messages,[{'EXIT',<0.22619.248>,connection_closed},
{'$gen_call',{<0.22.0>,#Ref<0.0.622.220265>},
{connect,normal,'mochisvn@REDACTED'}},
{'$gen_call',{<6341.14452.446>,#Ref<6341.0.542.13516>},
{is_auth,'discover@REDACTED'}},
tick,tick,tick,tick,
{'EXIT',<0.1348.260>,connection_closed},
{'EXIT',<0.679.0>,connection_closed},
{'EXIT',<0.6370.250>,connection_closed},
{'EXIT',<0.18818.245>,connection_closed},
{'EXIT',<0.22656.248>,connection_closed},
{'EXIT',<0.21639.321>,connection_closed},
{'EXIT',<0.612.0>,connection_closed},
{'EXIT',<0.12655.294>,connection_closed},
{'EXIT',<0.23280.101>,connection_closed},
{'EXIT',<0.183.0>,connection_closed},
{'EXIT',<0.673.0>,connection_closed},
{'EXIT',<0.18711.245>,...},
{'EXIT',...},
{...}|...]},
{links,[<0.12300.159>,<0.22603.248>,<0.11959.295>,
<0.1474.310>,<0.1479.310>,<0.30534.333>,<0.14155.304>,
<0.1794.308>,<0.8435.304>,<0.32206.259>,<0.11952.295>,
<0.11954.295>,<0.9573.266>,<0.1597.285>,<0.20340.259>,
<0.32169.259>,<0.19993.259>,<0.22511.248>,<0.22554.248>,
<0.22588.248>|...]},
{dictionary,[{'$ancestors',[net_sup,kernel_sup,<0.9.0>]},
{longnames,true},
{'$initial_call',{gen,init_it,
[gen_server,<0.19.0>,<0.19.0>,
{local,net_kernel},
net_kernel,
{'mochiadsdb@REDACTED',longnames,15000},
[]]}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,max},
{group_leader,<0.8.0>},
{total_heap_size,8024355},
{heap_size,8024355},
{stack_size,27},
{reductions,933092852},
{garbage_collection,[{fullsweep_after,65535},{minor_gcs,0}]},
{suspending,[]}]
More information about the erlang-bugs
mailing list