[erlang-bugs] net_kernel waiting for response from self

Matthew Dempsky <>
Wed Sep 17 01:43:54 CEST 2008


Earlier we noticed one of our Erlang nodes was not connected to a
bunch of other nodes, and that issuing net_adm:ping(Node) to try to
reconnect it would just hang.  It seems that net_kernel had issued a
gen_server:call to itself and was waiting indefinitely for a response.

Complete backtrace of Pid (net_kernel, <0.22.0>) is below.  From the
back trace you can see net_kernel made a call to

    gen_server:call(net_kernel, {connect, normal,
''}, infinity)

and sent the request with #Ref<0.0.622.220265>.

To confirm this, you can look at the process_info dump and see the
second element in the message queue is

            {'$gen_call',{<0.22.0>,#Ref<0.0.622.220265>},
                         {connect,normal,''}}.

I forged a response by grabbing that references and calling

    Pid ! {Ref, false},

which finally led net_kernel to proceed a little bit further, but it
immediately hung trying to handle

            {'$gen_call',{<6341.14452.446>,#Ref<6341.0.542.13516>},
                         {is_auth,''}}.

I repeated the process a few more times, forging responses to the
connect calls, but eventually gave up and just restarted the node
after determining there were 300,000 pending is_auth requests in the
message queue, and each needed a forged response to continue.


()88> bt(Pid).
Program counter: 0x00002aaaab841870 (gen:wait_resp_mon/3 + 64)
CP: 0x00002aaaab89dc30 (gen_server:call/3 + 160)
arity = 0

0x00002aaae3591870 Return addr 0x00002aaaab89dc30 (gen_server:call/3 + 160)
y(0)     infinity
y(1)     #Ref<0.0.622.220265>
y(2)     ''

0x00002aaae3591890 Return addr 0x00002aaaab812d88 (erlang:dsend/2 + 136)
y(0)     infinity
y(1)     {connect,normal,''}
y(2)     net_kernel
y(3)     Catch 0x00002aaaab89dc30 (gen_server:call/3 + 160)

0x00002aaae35918b8 Return addr 0x00002aaaab89e160 (gen_server:reply/2 + 208)
y(0)     {#Ref<6397.0.34.240428>,yes}
y(1)     <6397.6917.55>

0x00002aaae35918d0 Return addr 0x00002aaaab8a2370
(gen_server:handle_msg/5 + 848)
y(0)     Catch 0x00002aaaab89e160 (gen_server:reply/2 + 208)

0x00002aaae35918e0 Return addr 0x00002aaaab853310 (proc_lib:init_p/5 + 400)
y(0)     net_kernel
y(1)     []
y(2)     net_kernel
y(3)     <0.19.0>
y(4)     []
y(5)     []
y(6)     {state,'','',longnames,{tick,<0.24.0>,15000},7000,sys_dist,[{<0.27218.334>,''},{<0.5652.334>,''},{<0.30534.333>,''},{<0.14584.332>,''},{<0.10693.325>,''},{<0.10679.325>,''},{<0.8624.325>,''},{<0.21639.321>,''},{<0.21631.321>,''},{<0.1479.310>,''},{<0.1474.310>,''},{<0.1794.308>,''},{<0.14155.304>,''},{<0.8435.304>,''},{<0.11959.295>,''},{<0.11954.295>,''},{<0.11952.295>,''},{<0.12655.294>,''},{<0.1597.285>,''},{<0.31519.268>,''},{<0.31492.268>,''},{<0.31418.268>,''},{<0.28344.266>,''},{<0.28341.266>,''},{<0.9573.266>,''},{<0.28064.264>,''},{<0.24678.264>,''},{<0.25804.263>,''},{<0.4231.260>,''},{<0.1352.260>,''},{<0.1348.260>,''},{<0.1346.260>,''},{<0.32206.259>,''},{<0.32169.259>,''},{<0.20340.259>,''},{<0.19993.259>,''},{<0.16961.250>,''},{<0.6399.250>,''},{<0.6370.250>,''},{<0.6286.250>,''},{<0.6278.250>,''},{<0.22656.248>,''},{<0.22654.248>,''},{<0.22653.248>,''},{<0.22625.248>,''},{<0.22623.248>,''},{<0.22621.248>,''},{<0.22619.248>,''},{<0.22603.248>,''},{<0.22593.248>,''},{<0.22588.248>,''},{<0.22559.248>,''},{<0.22554.248>,''},{<0.22546.248>,''},{<0.22542.248>,''},{<0.22538.248>,''},{<0.22524.248>,''},{<0.22519.248>,''},{<0.22515.248>,''},{<0.22511.248>,''},{<0.22507.248>,''},{<0.22487.248>,''},{<0.22474.248>,''},{<0.22469.248>,''},{<0.19636.245>,''},{<0.18890.245>,''},{<0.18885.245>,''},{<0.18856.245>,''},{<0.18850.245>,''},{<0.18830.245>,''},{<0.18824.245>,''},{<0.18818.245>,''},{<0.18814.245>,''},{<0.18800.245>,''},{<0.18791.245>,''},{<0.18789.245>,''},{<0.18786.245>,''},{<0.18721.245>,''},{<0.18711.245>,''},{<0.18575.245>,''},{<0.18556.245>,''},{<0.18535.245>,''},{<0.18423.245>,''},{<0.12300.159>,''},{<0.23778.120>,''},{<0.3295.120>,''},{<0.23280.101>,''},{<0.23276.101>,''},{<0.3402.67>,''},{<0.3398.67>,''},{<0.3339.67>,''},{<0.3335.67>,''},{<0.3328.67>,''},{<0.8774.2>,''},{<0.673.0>,''},{<0.679.0>,''},{<0.612.0>,''},{<0.350.0>,''},{<0.342.0>,''},{<0.344.0>,''},{<0.333.0>,''},{<0.288.0>,''},{<0.227.0>,''},{<0.77.0>,''},{<0.183.0>,''},{<0.117.0>,''},{<0.103.0>,''},{<0.98.0>,''},{<0.91.0>,''},{<0.84.0>,''},{<0.75.0>,''},{<0.68.0>,''}],[],[{listen,#Port<0.7>,<0.23.0>,{net_address,{{0,0,0,0},39758},"reaver",tcp,inet},inet_tcp_dist}],[],0,all}

0x00002aaae3591920 Return addr 0x00000000008572f8 (<terminate process normally>)
y(0)     Catch 0x00002aaaab853330 (proc_lib:init_p/5 + 432)
y(1)     gen
y(2)     init_it
y(3)     [gen_server,<0.19.0>,<0.19.0>,{local,net_kernel},net_kernel,{'',longnames,15000},[]]
ok

()89> erlang:process_info(Pid).
[{registered_name,net_kernel},
 {current_function,{gen,wait_resp_mon,3}},
 {initial_call,{proc_lib,init_p,5}},
 {status,waiting},
 {message_queue_len,458946},
 {messages,[{'EXIT',<0.22619.248>,connection_closed},
            {'$gen_call',{<0.22.0>,#Ref<0.0.622.220265>},
                         {connect,normal,''}},
            {'$gen_call',{<6341.14452.446>,#Ref<6341.0.542.13516>},
                         {is_auth,''}},
            tick,tick,tick,tick,
            {'EXIT',<0.1348.260>,connection_closed},
            {'EXIT',<0.679.0>,connection_closed},
            {'EXIT',<0.6370.250>,connection_closed},
            {'EXIT',<0.18818.245>,connection_closed},
            {'EXIT',<0.22656.248>,connection_closed},
            {'EXIT',<0.21639.321>,connection_closed},
            {'EXIT',<0.612.0>,connection_closed},
            {'EXIT',<0.12655.294>,connection_closed},
            {'EXIT',<0.23280.101>,connection_closed},
            {'EXIT',<0.183.0>,connection_closed},
            {'EXIT',<0.673.0>,connection_closed},
            {'EXIT',<0.18711.245>,...},
            {'EXIT',...},
            {...}|...]},
 {links,[<0.12300.159>,<0.22603.248>,<0.11959.295>,
         <0.1474.310>,<0.1479.310>,<0.30534.333>,<0.14155.304>,
         <0.1794.308>,<0.8435.304>,<0.32206.259>,<0.11952.295>,
         <0.11954.295>,<0.9573.266>,<0.1597.285>,<0.20340.259>,
         <0.32169.259>,<0.19993.259>,<0.22511.248>,<0.22554.248>,
         <0.22588.248>|...]},
 {dictionary,[{'$ancestors',[net_sup,kernel_sup,<0.9.0>]},
              {longnames,true},
              {'$initial_call',{gen,init_it,
                                    [gen_server,<0.19.0>,<0.19.0>,
                                     {local,net_kernel},
                                     net_kernel,
                                     {'',longnames,15000},
                                     []]}}]},
 {trap_exit,true},
 {error_handler,error_handler},
 {priority,max},
 {group_leader,<0.8.0>},
 {total_heap_size,8024355},
 {heap_size,8024355},
 {stack_size,27},
 {reductions,933092852},
 {garbage_collection,[{fullsweep_after,65535},{minor_gcs,0}]},
 {suspending,[]}]



More information about the erlang-bugs mailing list