net_kernel hang, perhaps blocked by busy_dist_port race?

[Scott Lystig Fritchie]

Hi, all.  We've been bitten by a rather mysterious bug that has
disrupted Erlang message passing on roughly 10% of all nodes in a 100+
node cluster.  The same thing happened on 10 nodes within a 2-3 second
time window.  No further communication with the affected nodes via
Erlang message passing is possible.

I'm wondering if there's a possible race condition when two nodes A
and Z are communicating with each other, like this:

   1. Z makes a bunch of RPCs to A.
   2. A starts sending RPC replies to Z.
   3. Z decides to behave erratically, cause unknown.
   4. A's TCP connection to Z becomes "busy", probably because Z
      cannot or will not read data on the A <-> Z TCP connection.
   5. All processes on A that are trying to reply to Z are blocked and
      unscheduled; 'busy_dist_port' messages are generated for all of
      them. 
   6. The 'net_kernel' process on A is one of the procs blocked by the
      'busy_dist_port' events.
   7. A's connection to Z is broken.  The system message reported is:
      {nodedown_reason,connection_closed},{node_type,visible}

... and then A's 'net_kernel' process remains blocked forever?  Or is
alive but isn't working correctly?

Details below.  Sorry I don't have a patch available.

-Scott

Environment
-----------

    * ErlangOTP R13B04, -smp auto +A 64 +K true
        - Patched to change "erts_de_busy_limit" from default 
          128KB to 4096KB
    * Linux kernel, RedHat EL4 kernel IIRC of some flavor
    * NTP configured and running correctly on all machines (to help
      correlate log file timestamps)
    * Cluster of 50+ physical machines, 100+ Erlang VMs/nodes total
    * All nodes are using erlang:system_monitor() BIF for big
      heap, long garbage collection, and busy dist port events.
    * All nodes report node up and down events via BIF
      erlang:process_flag(monitor_nodes,true).

Sequence of events summary
--------------------------

1. One node in the cluster (checking the health of other nodes) makes
several thousand gen_server RPC calls to various servers on all other
nodes in somewhere between 1 and 5 second cycles (depending on what's
being monitored).  This node's name is 'app@REDACTED'.

2. The 'app@REDACTED' hits a weird problem.  We still can't figure out
what happened, but it behaved like an extremely intermittent network
partition that only affected boxZ.

3. Within 2-3 seconds, 10 nodes on the network become completely
unresponsive and cannot recover:

    * The 'app@REDACTED' node cannot talk to them, but that's not
      surprising because 'app@REDACTED' is having its own problems.
    * All other nodes report net_tick_timeout errors.
    * All attempts such as "erl -sname tmp$$ -remsh app@REDACTED" to
      connect to the hosed node fail.

Sequence of events detail
-------------------------

* At time T, +- 1 second, there are multiple reports of the same net
distribution port being blocked, e.g. #Port<0.213633546> on app@REDACTED:

  a. {monitor,<0.6824.3617>,busy_dist_port,#Port<0.213633546>}
     This is from the VM, triggered by the system_monitor BIF()

  b. sysmon_server: process <0.6824.3617> info:
     [{registered_name,foo},{initial_call,{proc_lib,init_p,5}},{current_function,{erlang,bif_return_trap,1}}]
     This is from the system_monitor event collector, which tries to
     find some helpful info about the process.

All 10 machines register anywhere from 8 to 15 of these pairs of
messages.  For each machine, all complaints are about the same Erlang
port #.

* Within T + 1 seconds, there's a report on the same port # that the
net_kernel process has been blocked, e.g. on app@REDACTED:

  a. {monitor,<0.23.0>,busy_dist_port,#Port<0.213633546>}

  b. sysmon_server: process <0.23.0> info:
     [{registered_name,net_kernel},{initial_call,{proc_lib,init_p,5}},{current_function,{erlang,bif_return_trap,1}}]

There is no direct evidence that shows that the blocked ports, e.g.
#Port<0.213633546> on app@REDACTED, are the ones that are used for
communication with app@REDACTED node, but it appears quite likely to be
true.

* Within T + 3 seconds (and usually within T + 2 seconds), there's a
  report that app@REDACTED is down, e.g. on app@REDACTED:

  net_kernel: node app@REDACTED down info [{nodedown_reason,connection_closed},{node_type,visible}]

* 20 seconds later, all other nodes in the cluster drop their connections
to these 9 nodes, due to {nodedown_reason,net_tick_timeout} reason.

* No further communication via Erlang message passing is possible:
  existing nodes cannot reconnect, and new nodes (e.g. "erl -remsh
  app@REDACTED") cannot connect.

* We used "gcore" to snag core dumps from 4 of the 10 affected nodes.
  The GDB backtrace doesn't reveal much to my untrained eyes.

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /lib64/libutil.so.1...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/libncurses.so.5...done.
Loaded symbols for /usr/lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /ert/lib/erlang/lib/crypto-1.6.4/priv/lib/crypto_drv.so...done.
Loaded symbols for /ert/lib/erlang/lib/crypto-1.6.4/priv/lib/crypto_drv.so
Reading symbols from /ert/openssl/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /ert/openssl/lib/libcrypto.so.0.9.8
Core was generated by `/ert/lib/erlang/erts-5.7.5/bin/beam.smp'.
[New process 18963]
[New process 18964]
[New process 18965]
[New process 18966]
[New process 18967]
[New process 18968]
[New process 18969]
[New process 18970]
[New process 18971]
[New process 18972]
[New process 18973]
[New process 18974]
[New process 18975]
[New process 18976]
[New process 18977]
[New process 18978]
[New process 18979]
[New process 18980]
[New process 18981]
[New process 18982]
[New process 18983]
[New process 18984]
[New process 18985]
[New process 18986]
[New process 18987]
[New process 18988]
[New process 18989]
[New process 18990]
[New process 18991]
[New process 18992]
[New process 18993]
[New process 18994]
[New process 18995]
[New process 18996]
[New process 18997]
[New process 18998]
[New process 18999]
[New process 19000]
[New process 19001]
[New process 19002]
[New process 19003]
[New process 19004]
[New process 19005]
[New process 19006]
[New process 19007]
[New process 19008]
[New process 19009]
[New process 19010]
[New process 19011]
[New process 19012]
[New process 19013]
[New process 19014]
[New process 19015]
[New process 19016]
[New process 19017]
[New process 19018]
[New process 19019]
[New process 19020]
[New process 19021]
[New process 19022]
[New process 19023]
[New process 19024]
[New process 19025]
[New process 19026]
[New process 19027]
[New process 19028]
[New process 19029]
[New process 19030]
[New process 19031]
[New process 19032]
[New process 19033]
[New process 19034]
[New process 19035]
[New process 19036]
[New process 19037]
[New process 19038]
[New process 19039]
[New process 19040]
[New process 19041]
[New process 19042]
[New process 19043]
[New process 19044]
[New process 19045]
[New process 18956]
#0  0x0000003383c0d2cb in read () from /lib64/libpthread.so.0
(gdb) thread apply all where

Thread 84 (process 18956):
#0  0x00000033830cc5e2 in select () from /lib64/libc.so.6
#1  0x000000000052a900 in erts_sys_main_thread () at sys/unix/sys.c:3019
#2  0x000000000044d1ef in erl_start (argc=35, argv=<value optimized out>) at beam/erl_init.c:1330
#3  0x0000000000430429 in main (argc=0, argv=0x0) at sys/unix/erl_main.c:29

Thread 83 (process 19045):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=16, rq=0x2b035e3bb3b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 82 (process 19044):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=15, rq=0x2b035e3bb1b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 81 (process 19043):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=14, rq=0x2b035e3bafb0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 80 (process 19042):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=13, rq=0x2b035e3badb0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 79 (process 19041):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=12, rq=0x2b035e3babb0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 78 (process 19040):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=11, rq=0x2b035e3ba9b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 77 (process 19039):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=10, rq=0x2b035e3ba7b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 76 (process 19038):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=9, rq=0x2b035e3ba5b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 75 (process 19037):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=8, rq=0x2b035e3ba3b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 74 (process 19036):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=7, rq=0x2b035e3ba1b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 73 (process 19035):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=6, rq=0x2b035e3b9fb0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 72 (process 19034):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=5, rq=0x2b035e3b9db0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 71 (process 19033):
#0  0x00000033830d3488 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000530f04 in erts_poll_wait_kp (ps=0x2b035e2f3d38, pr=0x44b25660, len=0x44b25e7c, utvp=<value optimized out>) at sys/common/erl_poll.c:1907
#2  0x0000000000533bdb in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1156
#3  0x000000000049b896 in sched_sys_wait (no=4, rq=0x2b035e3b9bb0) at beam/erl_process.c:785
#4  0x00000000004a1dd2 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6020
#5  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#6  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#7  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#8  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 70 (process 19032):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=3, rq=0x2b035e3b99b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 69 (process 19031):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=2, rq=0x2b035e3b97b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 68 (process 19030):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000049b6e2 in sched_cnd_wait (no=1, rq=0x2b035e3b95b0) at beam/erl_threads.h:632
#2  0x00000000004a1a93 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:6026
#3  0x000000000050eb2d in process_main () at beam/beam_emu.c:1161
#4  0x000000000049f322 in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3060
#5  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#6  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 67 (process 19029):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 66 (process 19028):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 65 (process 19027):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 64 (process 19026):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 63 (process 19025):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 62 (process 19024):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 61 (process 19023):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 60 (process 19022):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 59 (process 19021):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 58 (process 19020):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 57 (process 19019):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 56 (process 19018):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 55 (process 19017):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 54 (process 19016):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 53 (process 19015):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 52 (process 19014):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 51 (process 19013):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 50 (process 19012):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 49 (process 19011):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 48 (process 19010):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 47 (process 19009):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 46 (process 19008):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 45 (process 19007):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 44 (process 19006):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 43 (process 19005):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 42 (process 19004):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 41 (process 19003):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 40 (process 19002):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 39 (process 19001):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 38 (process 19000):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 37 (process 18999):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 36 (process 18998):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 35 (process 18997):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 34 (process 18996):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 33 (process 18995):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 32 (process 18994):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 31 (process 18993):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 30 (process 18992):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 29 (process 18991):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 28 (process 18990):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 27 (process 18989):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 26 (process 18988):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 25 (process 18987):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 24 (process 18986):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 23 (process 18985):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 22 (process 18984):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 21 (process 18983):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 20 (process 18982):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 19 (process 18981):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 18 (process 18980):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 17 (process 18979):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 16 (process 18978):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 15 (process 18977):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 14 (process 18976):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 13 (process 18975):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 12 (process 18974):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 11 (process 18973):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 10 (process 18972):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 9 (process 18971):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 8 (process 18970):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 7 (process 18969):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 6 (process 18968):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 5 (process 18967):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 4 (process 18966):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004e9bdf in async_main (arg=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 3 (process 18965):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000046df1c in sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 2 (process 18964):
#0  0x0000003383c0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000471ee4 in emergency_watchdog (unused=<value optimized out>) at beam/erl_threads.h:632
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6

Thread 1 (process 18963):
#0  0x0000003383c0d2cb in read () from /lib64/libpthread.so.0
#1  0x000000000052b51e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2913
#2  0x0000000000585064 in thr_wrapper (vtwd=<value optimized out>) at common/ethread.c:475
#3  0x0000003383c06367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033830d309d in clone () from /lib64/libc.so.6
(gdb)