[erlang-questions] net_kernel fails to start (nodistribution)

[Garry Hodgson]

problem solved. it turns out that it was due to a faulty
iptables rule set.  whew!

On 10/01/2015 12:05 PM, Garry Hodgson wrote:
> I'm seeing something odd on one of the openstack VM's
> we're using.  We've been using this VM for some time with no
> problems, but all of the sudden I can't run erlang with
> distribution. Running just "erl" is fine, but if I specify an
> sname it hangs for minutes, then fails with crash dump
> (and output at end of message).
>
> I've restarted epmd, and run it in debug mode, but don't
> see anything amiss. We did make some changes yesterday
> to the OpenStack security groups for the VM, which may be
> related (ports 80, 443, 2121 and 62201).  Hostname, ifconfig,
> and iptables rules all seem ok.
>
> Looking through the code, it appears the failure occurs in
> net_kernel:init_node(Name, LongOrShortNames), or maybe
> further down in start_protos( Name, Node ).
>
> I've never seen anything like this, and google only shows me
> a few rabbit mq questions that didn't help much.
> Any insight would be appreciated.
>
> Thanks
>
> example output:
>
> # erl -sname foo
> {error_logger,{{2015,10,1},{14,58,47}},"Protocol: ~tp: register/listen 
> error: ~tp~n",["inet_tcp",etimedout]}
> {error_logger,{{2015,10,1},{14,58,47}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,322}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[#Port<0.54>,<0.18.0>]},{dictionary,[{longnames,false}]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,739}],[]]} 
>
> {error_logger,{{2015,10,1},{14,58,47}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},{mfargs,{net_kernel,start_link,[[foo,shortnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} 
>
> {error_logger,{{2015,10,1},{14,58,47}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} 
>
> {error_logger,{{2015,10,1},{14,58,47}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]} 
>
> {error_logger,{{2015,10,1},{14,58,47}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]} 
>
> {"Kernel pid 
> terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}}"}
>
> Crash dump was written to: erl_crash.dump
> Kernel pid terminated (application_controller) 
> ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k
>


-- 
Garry Hodgson
Lead Member of Technical Staff
AT&T Chief Security Office (CSO)

"This e-mail and any files transmitted with it are AT&T property, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited."