[erlang-questions] exception error: a system limit has been reached

[Khitai Pang]

When I restart my app I get "escript: exception error: a system limit 
has been reached".  I can't figure out the reason.


$ myapp/_rel/myapp/bin/myapp restart

Crash dump is being written to: erl_crash.dump...done
Kernel pid terminated (application_controller) 
({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k
escript: exception error: a system limit has been reached
   in function  list_to_atom/1
      called as 
list_to_atom("myapp_maint_5500@{error_logger,{{2016,5,8},{12,24,32}},\"Protocol: 
~tp: register/listen error: 
~tp~n\",[\"inet_tcp\",econnrefused]}\r\n{error_logger,{{2016,5,8},{12,24,32}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,<0.22.0>},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,344}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[#Port<0.195>,<0.19.0>]},{dictionary,[{longnames,true}]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,868}],[]]}\r\n{error_logger,{{2016,5,8},{12,24,32}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{id,net_kernel},{mfargs,{net_kernel,start_link,[['longname910b8272-myapp',longnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]}\r\n{error_logger,{{2016,5,8},{12,24,32}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}},{offender,[{pid,undefined},{id,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}\r\n{error_logger,{{2016,5,8},{12,24,32}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,\"application_master.erl\"},{line,134}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]}\r\n{error_logger,{{2016,5,8},{12,24,32}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]}\r\n{\"Kernel 
pid 
terminated\",application_controller,\"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}}\"}\r")


It's Ubuntu 14.04.4 x86_64 running in a cloud vm with 2G memory.

I have the following in /etc/sysctl.conf:

## File Handle Limits

# Increate the maximum number of open files
fs.file-max = 500000

## Socket Tuning

# Use the full range of ports
net.ipv4.ip_local_port_range = 1024 65535
# Enables fast recycling of TIME_WAIT sockets
net.ipv4.tcp_tw_recycle = 1
# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1
# Increase the maximum socket receive/send buffer size to 16MB
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# Enable SYN cookies
net.ipv4.tcp_syncookies = 1
# Increase the number of outstanding syn requests allowed
net.ipv4.tcp_max_syn_backlog = 4096
# Increase the maximum number of requests queued to a listen socket
net.core.somaxconn = 1024
# Increase the maximum number of packets queued on the INPUT side
net.core.netdev_max_backlog = 4096
# TCP receive/send buffer size [min, default, max]
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.core.somaxconn = 50000


And the following in /etc/security/limits.conf:

root soft nofile 65535
root hard nofile 65535
* soft nofile 65535
* hard nofile 65535

free output:

 > free
              total       used       free     shared buffers     cached
Mem:       2047276     979708    1067568        480 56272     264548
-/+ buffers/cache:     658888    1388388
Swap:            0          0          0


Any idea what system limit it reaches and why?


Thanks
Khitai