[erlang-questions] crash dump at ejabberd startup

Wed Nov 17 04:59:52 CET 2010

> It might help. I think the quickest way to debug this would be to:

donesults below ...

> 1. Start up epmd manually in the jail with the debug switches
> 
> epmd -d -d -d

I did that already before but repeated it nonetheless.

# epmd -d -d -d
epmd: Wed Nov 17 03:27:57 2010: epmd running - daemon = 0
epmd: Wed Nov 17 03:27:57 2010: try to initiate listening port 4369
epmd: Wed Nov 17 03:27:57 2010: starting
epmd: Wed Nov 17 03:27:57 2010: entering the main select() loop
epmd: Wed Nov 17 03:28:02 2010: time in seconds: 1289964482
epmd: Wed Nov 17 03:28:07 2010: time in seconds: 1289964487
epmd: Wed Nov 17 03:28:12 2010: time in seconds: 1289964492
epmd: Wed Nov 17 03:28:17 2010: time in seconds: 1289964497
epmd: Wed Nov 17 03:28:22 2010: time in seconds: 1289964502
^C

You before mentioned, it did not even try to connect
(can''t say anything about this).

> 2. Run tcpdump on port 4369 for the interfaces in the jail

# tcpdump  -vv port 4369
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 96 
bytes

tcpdump doesn't get anything when listening on the em0 interface. 
So, I tried 

# tcpdump -i lo0 -vv port 4369
tcpdump: WARNING: lo0: no IPv4 address assigned
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 96 
bytes

> 3. Start up a distributed erlang node by hand:
> 
> erl -name foo

# erl -name foo
{error_logger,{{2010,11,17},{3,33,15}},"Protocol: ~p: register error: ~p~n",
["inet_tcp",{{badmatch,{error,epmd_close}},[{inet_tcp_dist,listen,1},
{net_kernel,start_protos,4},{net_kernel,start_protos,3},
{net_kernel,init_node,2},{net_kernel,init,1},{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}]}
{error_logger,{{2010,11,17},{3,33,15}},crash_report,[[{initial_call,
{net_kernel,init,['Argument__1']}},{pid,<0.19.0>},{registered_name,[]},
{error_info,{exit,{error,badarg},[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},{ancestors,[net_sup,kernel_sup,<0.9.0>]},
{messages,[]},{links,[#Port<0.56>,<0.16.0>]},{dictionary,
[{longnames,true}]},{trap_exit,true},{status,running},{heap_size,377},
{stack_size,24},{reductions,443}],[]]}
{error_logger,{{2010,11,17},{3,33,15}},supervisor_report,[{supervisor,
{local,net_sup}},{errorContext,start_error},{reason,
{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},
{mfargs,{net_kernel,start_link,[[foo,longnames]]}},{restart_type,permanent},
{shutdown,2000},{child_type,worker}]}]}
{error_logger,{{2010,11,17},{3,33,15}},supervisor_report,[{supervisor,
{local,kernel_sup}},{errorContext,start_error},{reason,shutdown},{offender,
[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},
{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2010,11,17},{3,33,15}},std_info,[{application,kernel},
{exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid 
terminated",application_controller,"{application_start_failure,kernel,
{shutdown,{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) 
({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})

and tcpdump output:

03:33:15.964680 IP (tos 0x0, ttl 64, id 24389, offset 0, flags [DF], proto 
TCP (6), length 60, bad cksum 0 (->4f37)!)
    mail.kepos.org.10975 > mail.kepos.org.4369: Flags [S], cksum 0xd692 
(correct), seq 3123827793, win 65535, options [mss 16344,nop,wscale 
3,sackOK,TS val 77847542 ecr 0], length 0
03:33:15.964701 IP (tos 0x0, ttl 64, id 24390, offset 0, flags [DF], proto 
TCP (6), length 60, bad cksum 0 (->4f36)!)
    mail.kepos.org.4369 > mail.kepos.org.10975: Flags [S.], cksum 0x776a 
(correct), seq 3252621940, ack 3123827794, win 65535, options [mss 
16344,nop,wscale 3,sackOK,TS val 2304704868 ecr 77847542], length 0
03:33:15.964712 IP (tos 0x0, ttl 64, id 24391, offset 0, flags [DF], proto 
TCP (6), length 52, bad cksum 0 (->4f3d)!)
    mail.kepos.org.10975 > mail.kepos.org.4369: Flags [.], cksum 0xbd56 
(correct), seq 1, ack 1, win 8960, options [nop,nop,TS val 77847542 ecr 
2304704868], length 0
03:33:15.964740 IP (tos 0x0, ttl 64, id 24392, offset 0, flags [DF], proto 
TCP (6), length 70, bad cksum 0 (->4f2a)!)
    mail.kepos.org.10975 > mail.kepos.org.4369: Flags [P.], cksum 0x6619 
(correct), seq 1:19, ack 1, win 8960, options [nop,nop,TS val 77847542 ecr 
2304704868], length 18
03:33:15.964896 IP (tos 0x0, ttl 64, id 24393, offset 0, flags [DF], proto 
TCP (6), length 52, bad cksum 0 (->4f3b)!)
    mail.kepos.org.4369 > mail.kepos.org.10975: Flags [F.], cksum 0xbd43 
(correct), seq 1, ack 19, win 8960, options [nop,nop,TS val 2304704868 ecr 
77847542], length 0
03:33:15.964906 IP (tos 0x0, ttl 64, id 24394, offset 0, flags [DF], proto 
TCP (6), length 52, bad cksum 0 (->4f3a)!)
    mail.kepos.org.10975 > mail.kepos.org.4369: Flags [.], cksum 0xbd43 
(correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 ecr 
2304704868], length 0
03:33:15.964929 IP (tos 0x0, ttl 64, id 24395, offset 0, flags [DF], proto 
TCP (6), length 52, bad cksum 0 (->4f39)!)
    mail.kepos.org.10975 > mail.kepos.org.4369: Flags [F.], cksum 0xbd42 
(correct), seq 19, ack 2, win 8960, options [nop,nop,TS val 77847542 ecr 
2304704868], length 0
03:33:15.964937 IP (tos 0x0, ttl 64, id 24396, offset 0, flags [DF], proto 
TCP (6), length 52, bad cksum 0 (->4f38)!)
    mail.kepos.org.4369 > mail.kepos.org.10975: Flags [.], cksum 0xbd43 
(correct), seq 2, ack 20, win 8959, options [nop,nop,TS val 2304704868 ecr 
77847542], length 0
^C
8 packets captured
41 packets received by filter
0 packets dropped by kernel

Juust to exde any misconfiguation on the machine I allso  tried itt again 
switching off the firewall for this test: same result.

> Report back on what you find!

I fear there's no real enlightenment from this - at least not for me. We 
already knew that we cannot make full use of the loopback interface 
regarding the virtualization situation in a FreeBSD jail.

Also, I still do not know what actually it is, Erlang does not like with 
it's environment.

Nonetheless there must be a way to get it up as there are people out there 
capable to do it. 

I'd like to again ask whether there was no way to make Erlang talk with the 
actual IP instead off talking to the localhost. 

And stll I'm unsure on how to deal with the inetrc file. What I read from 
your link (Michael) I find a bit unsual and confusing.

kind regards
Tom