R13B02 on 8/16 core box: all TCP communication hangs/frozen

Scott Lystig Fritchie fritchie@REDACTED
Wed Nov 18 13:16:01 CET 2009


Hi, all.  I've scoured the erlang-patches, erlang-bugs, and
erlang-questions mailing list archives for help on this, but it doesn't
seem to have popped up yet?

While running a distributed Erlang app on a box that probably wasn't
very busy at the time, all of its peer nodes decided that the hung box
had timed out (some whitespace and newlines flattened):

    ** Node foo@REDACTED not responding **** Removing (timedout) connection **

After running time of about 2 days, the VM appeared to lock up
completely.  Not possible to connect a remote shell.  No "strace -f -p"
output when connecting to the Erlang distribution TCP port reported by
"epmd -names", nor any output when connecting to any of the TCP listener
sockets owned by our Erlang applications inside the VM (connections via
Telnet or "nc" would open in 0-10 seconds, usually, because most app
listener sockets use a backlog size of 4096, but no sign of system call
activity by the VM).

The box in question is a few-months-old Dell (I can look up the model
number if it's helpful), dual CPU, quad core + hyperthreading (or
whatever Intel calls it today) for 16 virtual cores.  If I start an
Erlang shell without args:

    Erlang R13B02 (erts-5.7.3) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]
    
    Eshell V5.7.3  (abort with ^G)
    (slf21203@REDACTED)1> 
    
The app is run via:

    env ERL_MAX_ETS_TABLES=10007 \
            erl \
            +A 64 +K true \
            -noinput -noshell \
            -kernel dist_auto_connect once \
            -kernel net_ticktime 20 \
            ...

I've got output from "strace", "lsof", and a little bit of GDB stack
backtraces from each Pthread.  I've got a core file of the process via
"gcore", so I can do other GDB stuff if it'd be helpful.

I will almost certainly have at least 8 metric tons of extremely unhappy
customer asking lots of anxious questions within 12-24 hours.  Unhappy
me.  If there's other info that I can provide to the OTP team or other
debugging parties, please let me know ASAP.  And many thanks in advance!

-Scott

Output from "strace -f":

[pid 28271] futex(0x2b3519c8d404, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28270] futex(0x2b3519c8d204, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28269] futex(0x2b3519c8d004, FUTEX_WAIT_PRIVATE, 201, NULL <unfinished ...>
[pid 28268] futex(0x2b3519c8ce04, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28267] futex(0x2b3519c8cc04, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28266] futex(0x2b3519c8ca04, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28265] futex(0x2b3519c8c804, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28264] futex(0x2b3519c8c604, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28263] futex(0x2b3519c8c404, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28262] futex(0x2b3519c8c204, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28261] futex(0x2b3519c8c004, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28260] futex(0x2b3519c8be04, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28259] futex(0x2b3519c8bc04, FUTEX_WAIT_PRIVATE, 783259, NULL <unfinished ...>
[pid 28258] futex(0x2b3519c8ba04, FUTEX_WAIT_PRIVATE, 4212125, NULL <unfinished ...>
[pid 28257] futex(0x2b3519c8b804, FUTEX_WAIT_PRIVATE, 11335295, NULL <unfinished ...>
[pid 28256] futex(0x2b3519c8b604, FUTEX_WAIT_PRIVATE, 11458135, NULL <unfinished ...>
[pid 28255] wait4(-1,  <unfinished ...>
[pid 28254] futex(0x2aaaac92e3f4, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28253] futex(0x2aaaac92e364, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28252] futex(0x2aaaac92e2d4, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28251] futex(0x2aaaac92e244, FUTEX_WAIT_PRIVATE, 845, NULL <unfinished ...>
[pid 28250] futex(0x2aaaac92e1b4, FUTEX_WAIT_PRIVATE, 845, NULL <unfinished ...>
[pid 28249] futex(0x2aaaac92e124, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28248] futex(0x2aaaac92e094, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28247] futex(0x2aaaac92e004, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28246] futex(0x2aaaac92df74, FUTEX_WAIT_PRIVATE, 197345, NULL <unfinished ...>
[pid 28245] futex(0x2aaaac92dee4, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28244] futex(0x2aaaac92de54, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28243] futex(0x2aaaac92ddc4, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28242] futex(0x2aaaac92dd34, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28241] futex(0x2aaaac92dca4, FUTEX_WAIT_PRIVATE, 315, NULL <unfinished ...>
[pid 28240] futex(0x2aaaac92dc14, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28239] futex(0x2aaaac92db84, FUTEX_WAIT_PRIVATE, 152779, NULL <unfinished ...>
[pid 28238] futex(0x2aaaac92daf4, FUTEX_WAIT_PRIVATE, 121, NULL <unfinished ...>
[pid 28237] futex(0x2aaaac92da64, FUTEX_WAIT_PRIVATE, 121, NULL <unfinished ...>
[pid 28236] futex(0x2aaaac92d9d4, FUTEX_WAIT_PRIVATE, 121, NULL <unfinished ...>
[pid 28235] futex(0x2aaaac92d944, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28234] futex(0x2aaaac92d8b4, FUTEX_WAIT_PRIVATE, 1685, NULL <unfinished ...>
[pid 28233] futex(0x2aaaac92d824, FUTEX_WAIT_PRIVATE, 152827, NULL <unfinished ...>
[pid 28232] futex(0x2aaaac92d794, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28231] futex(0x2aaaac92d704, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28230] futex(0x2aaaac92d674, FUTEX_WAIT_PRIVATE, 119, NULL <unfinished ...>
[pid 28229] futex(0x2aaaac92d5e4, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28228] futex(0x2aaaac92d554, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28227] futex(0x2aaaac92d4c4, FUTEX_WAIT_PRIVATE, 783, NULL <unfinished ...>
[pid 28226] futex(0x2aaaac92d434, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28225] futex(0x2aaaac92d3a4, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28224] futex(0x2aaaac92d314, FUTEX_WAIT_PRIVATE, 158751, NULL <unfinished ...>
[pid 28223] futex(0x2aaaac92d284, FUTEX_WAIT_PRIVATE, 131, NULL <unfinished ...>
[pid 28222] futex(0x2aaaac92d1f4, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28221] futex(0x2aaaac92d164, FUTEX_WAIT_PRIVATE, 131, NULL <unfinished ...>
[pid 28220] futex(0x2aaaac92d0d4, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28219] futex(0x2aaaac92d044, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28218] futex(0x2aaaac92cfb4, FUTEX_WAIT_PRIVATE, 779, NULL <unfinished ...>
[pid 28217] futex(0x2aaaac92cf24, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28216] futex(0x2aaaac92ce94, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28215] futex(0x2aaaac92ce04, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28214] futex(0x2aaaac92cd74, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28213] futex(0x2aaaac92cce4, FUTEX_WAIT_PRIVATE, 121, NULL <unfinished ...>
[pid 28212] futex(0x2aaaac92cc54, FUTEX_WAIT_PRIVATE, 713, NULL <unfinished ...>
[pid 28211] futex(0x2aaaac92cbc4, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28210] futex(0x2aaaac92cb34, FUTEX_WAIT_PRIVATE, 795, NULL <unfinished ...>
[pid 28209] futex(0x2aaaac92caa4, FUTEX_WAIT_PRIVATE, 139, NULL <unfinished ...>
[pid 28208] futex(0x2aaaac92ca14, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28207] futex(0x2aaaac92c984, FUTEX_WAIT_PRIVATE, 121, NULL <unfinished ...>
[pid 28206] futex(0x2aaaac92c8f4, FUTEX_WAIT_PRIVATE, 123, NULL <unfinished ...>
[pid 28205] futex(0x2aaaac92c864, FUTEX_WAIT_PRIVATE, 121, NULL <unfinished ...>
[pid 28204] futex(0x2aaaac92c7d4, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28203] futex(0x2aaaac92c744, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28202] futex(0x2aaaac92c6b4, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28201] futex(0x2aaaac92c624, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28200] futex(0x2aaaac92c594, FUTEX_WAIT_PRIVATE, 133, NULL <unfinished ...>
[pid 28199] futex(0x2aaaac92c504, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28198] futex(0x2aaaac92c474, FUTEX_WAIT_PRIVATE, 139, NULL <unfinished ...>
[pid 28197] futex(0x2aaaac92c3e4, FUTEX_WAIT_PRIVATE, 1881, NULL <unfinished ...>
[pid 28196] futex(0x2aaaac92c354, FUTEX_WAIT_PRIVATE, 131, NULL <unfinished ...>
[pid 28195] futex(0x2aaaac92c2c4, FUTEX_WAIT_PRIVATE, 131, NULL <unfinished ...>
[pid 28194] futex(0x2aaaac92c234, FUTEX_WAIT_PRIVATE, 131, NULL <unfinished ...>
[pid 28193] futex(0x2aaaac92c1a4, FUTEX_WAIT_PRIVATE, 127, NULL <unfinished ...>
[pid 28192] futex(0x2aaaac92c114, FUTEX_WAIT_PRIVATE, 125, NULL <unfinished ...>
[pid 28191] futex(0x2aaaac92c084, FUTEX_WAIT_PRIVATE, 129, NULL <unfinished ...>
[pid 28190] futex(0x810ca4, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid 28189] read(6, [SLF: wait forever....]

Output from "lsof -nP"

COMMAND    PID   USER   FD   TYPE   DEVICE     SIZE     NODE NAME
beam.smp 28182 mmssys  cwd    DIR      8,3     4096 10485761 /m2h_data/data
beam.smp 28182 mmssys  rtd    DIR      8,1     4096        2 /
beam.smp 28182 mmssys  txt    REG      8,1 10282072 90608848 /usr/local/gemini/e
beam.smp 28182 mmssys  mem    REG      8,1   139416 48660505 /lib64/ld-2.5.so
beam.smp 28182 mmssys  mem    REG      8,1  1713096 48660507 /lib64/libc-2.5.so
beam.smp 28182 mmssys  mem    REG      8,1    23360 48660688 /lib64/libdl-2.5.so
beam.smp 28182 mmssys  mem    REG      8,1   615136 48660690 /lib64/libm-2.5.so
beam.smp 28182 mmssys  mem    REG      8,1   145592 48660704 /lib64/libpthread-2
beam.smp 28182 mmssys  mem    REG      8,1    53448 48660706 /lib64/librt-2.5.so
beam.smp 28182 mmssys  mem    REG      8,1    18152 48660804 /lib64/libutil-2.5.
beam.smp 28182 mmssys  mem    REG      8,1   380464 89141483 /usr/lib64/libncurs
beam.smp 28182 mmssys  mem    REG      8,1    77360 90605001 /usr/local/gemini/e
beam.smp 28182 mmssys  mem    REG      8,1  1781932 90608896 /usr/local/gemini/e
beam.smp 28182 mmssys    0r   CHR      1,3              3898 /dev/null
beam.smp 28182 mmssys    1w   CHR      1,3              3898 /dev/null
beam.smp 28182 mmssys    2w   CHR      1,3              3898 /dev/null
beam.smp 28182 mmssys    3r  0000     0,11        0 98551559 eventpoll
beam.smp 28182 mmssys    4r  FIFO      0,6          98551560 pipe
beam.smp 28182 mmssys    5w  FIFO      0,6          98551560 pipe
beam.smp 28182 mmssys    6r  FIFO      0,6          98551561 pipe
beam.smp 28182 mmssys    7w  FIFO      0,6          98551561 pipe
beam.smp 28182 mmssys    8u  IPv4 98551569               TCP *:45315 (LISTEN)
beam.smp 28182 mmssys    9u  IPv4 98551571               TCP 127.0.0.1:56642->12
beam.smp 28182 mmssys   10r  FIFO      0,6          98551574 pipe
beam.smp 28182 mmssys   11w   REG      8,6 36288720 21397512 /foo_log/log/archiv
beam.smp 28182 mmssys   12u  IPv4 98551581               TCP *:7585 (LISTEN)
beam.smp 28182 mmssys   13w  FIFO      0,6          98551575 pipe
beam.smp 28182 mmssys   14u  IPv4 98551584               TCP *:7576 (LISTEN)
beam.smp 28182 mmssys   15u  IPv4 98551587               TCP A.B.C.140:45315
beam.smp 28182 mmssys   16u  IPv4 98551605               TCP A.B.C.140:56270
beam.smp 28182 mmssys   17u  IPv4 98551606               TCP A.B.C.140:43152
beam.smp 28182 mmssys   18u  IPv4 98551608               TCP A.B.C.140:53789
beam.smp 28182 mmssys   19u  IPv4 98551607               TCP A.B.C.140:42325
beam.smp 28182 mmssys   20u  IPv4 98551609               TCP A.B.C.140:2958-
beam.smp 28182 mmssys   21u  IPv4 98551620               TCP A.B.C.140:55959
beam.smp 28182 mmssys   22u  sock      0,5          98697328 can't identify prot
beam.smp 28182 mmssys   23u  sock      0,5          98696776 can't identify prot
beam.smp 28182 mmssys   24u  sock      0,5          98698710 can't identify prot
beam.smp 28182 mmssys   25u  sock      0,5          98695296 can't identify prot
beam.smp 28182 mmssys   26u  sock      0,5          98698083 can't identify prot
beam.smp 28182 mmssys   27u  sock      0,5          98698783 can't identify prot
beam.smp 28182 mmssys   28u  sock      0,5          98698460 can't identify prot
beam.smp 28182 mmssys   30u  sock      0,5          98698635 can't identify prot
beam.smp 28182 mmssys   31u  sock      0,5          98685423 can't identify prot
beam.smp 28182 mmssys   32u  sock      0,5          98698786 can't identify prot
beam.smp 28182 mmssys   33u  sock      0,5          98698648 can't identify prot
beam.smp 28182 mmssys   34u  sock      0,5          98698717 can't identify prot
beam.smp 28182 mmssys   36u  sock      0,5          98686061 can't identify prot
beam.smp 28182 mmssys   37u  sock      0,5          98698545 can't identify prot
beam.smp 28182 mmssys   38u  sock      0,5          98686092 can't identify prot
beam.smp 28182 mmssys   41u  sock      0,5          98698407 can't identify prot
beam.smp 28182 mmssys   42u  sock      0,5          98698722 can't identify prot
beam.smp 28182 mmssys   43u  sock      0,5          98698729 can't identify prot

Output from GDB:

# gdb /usr/local/gemini/ert/R13B02/lib/erlang/erts-5.7.3/bin/beam.smp ./beam.smpfoo-Nov18.0357orso.core.28182 
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /lib64/libutil.so.1...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/libncurses.so.5...done.
Loaded symbols for /usr/lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/gemini/ert/R13B02/lib/erlang/lib/crypto-1.6.1/priv/lib/crypto_drv.so...done.
Loaded symbols for /usr/local/gemini/ert/R13B02/lib/erlang/lib/crypto-1.6.1/priv/lib/crypto_drv.so
Reading symbols from /usr/local/gemini/ert/R13B02/openssl/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/local/gemini/ert/R13B02/openssl/lib/libcrypto.so.0.9.8
Core was generated by `/usr/local/gemini/ert/R13B02/lib/erlang/erts-5.7.3/bin/beam.smp'.
[New process 28189]
[New process 28190]
[New process 28191]
[New process 28192]
[New process 28193]
[New process 28194]
[New process 28195]
[New process 28196]
[New process 28197]
[New process 28198]
[New process 28199]
[New process 28200]
[New process 28201]
[New process 28202]
[New process 28203]
[New process 28204]
[New process 28205]
[New process 28206]
[New process 28207]
[New process 28208]
[New process 28209]
[New process 28210]
[New process 28211]
[New process 28212]
[New process 28213]
[New process 28214]
[New process 28215]
[New process 28216]
[New process 28217]
[New process 28218]
[New process 28219]
[New process 28220]
[New process 28221]
[New process 28222]
[New process 28223]
[New process 28224]
[New process 28225]
[New process 28226]
[New process 28227]
[New process 28228]
[New process 28229]
[New process 28230]
[New process 28231]
[New process 28232]
[New process 28233]
[New process 28234]
[New process 28235]
[New process 28236]
[New process 28237]
[New process 28238]
[New process 28239]
[New process 28240]
[New process 28241]
[New process 28242]
[New process 28243]
[New process 28244]
[New process 28245]
[New process 28246]
[New process 28247]
[New process 28248]
[New process 28249]
[New process 28250]
[New process 28251]
[New process 28252]
[New process 28253]
[New process 28254]
[New process 28255]
[New process 28256]
[New process 28257]
[New process 28258]
[New process 28259]
[New process 28260]
[New process 28261]
[New process 28262]
[New process 28263]
[New process 28264]
[New process 28265]
[New process 28266]
[New process 28267]
[New process 28268]
[New process 28269]
[New process 28270]
[New process 28271]
[New process 28182]
#0  0x0000003d05a0d2cb in read () from /lib64/libpthread.so.0
(gdb) thread apply all info stack

Thread 84 (process 28182):
#0  0x0000003d04ecc5e2 in select () from /lib64/libc.so.6
#1  0x0000000000523290 in erts_sys_main_thread () at sys/unix/sys.c:3019
#2  0x000000000044bb7f in erl_start (argc=34, argv=<value optimized out>)
    at beam/erl_init.c:1275
#3  0x000000000042ee09 in main (argc=0, argv=0x0) at sys/unix/erl_main.c:29

Thread 83 (process 28271):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=16, rq=0x2b3519c8d3b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 82 (process 28270):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=15, rq=0x2b3519c8d1b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 81 (process 28269):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=14, rq=0x2b3519c8cfb0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
---Type <return> to continue, or q <return> to quit---
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 80 (process 28268):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=13, rq=0x2b3519c8cdb0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 79 (process 28267):
---Type <return> to continue, or q <return> to quit---
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=12, rq=0x2b3519c8cbb0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 78 (process 28266):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=11, rq=0x2b3519c8c9b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 77 (process 28265):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=10, rq=0x2b3519c8c7b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 76 (process 28264):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
---Type <return> to continue, or q <return> to quit---
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=9, rq=0x2b3519c8c5b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 75 (process 28263):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=8, rq=0x2b3519c8c3b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
---Type <return> to continue, or q <return> to quit---
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 74 (process 28262):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=7, rq=0x2b3519c8c1b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 73 (process 28261):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x0000000000494ab2 in sched_cnd_wait (no=6, rq=0x2b3519c8bfb0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 72 (process 28260):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=5, rq=0x2b3519c8bdb0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 71 (process 28259):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=4, rq=0x2b3519c8bbb0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 70 (process 28258):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=3, rq=0x2b3519c8b9b0)
---Type <return> to continue, or q <return> to quit---
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 69 (process 28257):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=2, rq=0x2b3519c8b7b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
---Type <return> to continue, or q <return> to quit---
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 68 (process 28256):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000494ab2 in sched_cnd_wait (no=1, rq=0x2b3519c8b5b0)
    at beam/erl_threads.h:634
#2  0x000000000049f4c3 in schedule (p=<value optimized out>, 
    calls=<value optimized out>) at beam/erl_process.c:5992
#3  0x0000000000507bcd in process_main () at beam/beam_emu.c:1126
#4  0x000000000049fd06 in sched_thread_func (vesdp=<value optimized out>)
    at beam/erl_process.c:3045
#5  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#6  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 67 (process 28255):
#0  0x0000003d05a0e03f in waitpid () from /lib64/libpthread.so.0
#1  0x000000000052448e in child_waiter (unused=<value optimized out>)
    at sys/unix/sys.c:2828
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 66 (process 28254):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 65 (process 28253):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 64 (process 28252):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 63 (process 28251):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 62 (process 28250):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
---Type <return> to continue, or q <return> to quit---
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 61 (process 28249):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 60 (process 28248):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
---Type <return> to continue, or q <return> to quit---
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 59 (process 28247):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 58 (process 28246):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 57 (process 28245):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 56 (process 28244):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 55 (process 28243):
---Type <return> to continue, or q <return> to quit---
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 54 (process 28242):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 53 (process 28241):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 52 (process 28240):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 51 (process 28239):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
---Type <return> to continue, or q <return> to quit---
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 50 (process 28238):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 49 (process 28237):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 48 (process 28236):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 47 (process 28235):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 46 (process 28234):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 45 (process 28233):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 44 (process 28232):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 43 (process 28231):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 42 (process 28230):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 41 (process 28229):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 40 (process 28228):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 39 (process 28227):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
---Type <return> to continue, or q <return> to quit---
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 38 (process 28226):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 37 (process 28225):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
---Type <return> to continue, or q <return> to quit---
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 36 (process 28224):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 35 (process 28223):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 34 (process 28222):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 33 (process 28221):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 32 (process 28220):
---Type <return> to continue, or q <return> to quit---
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 31 (process 28219):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 30 (process 28218):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 29 (process 28217):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 28 (process 28216):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
---Type <return> to continue, or q <return> to quit---
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 27 (process 28215):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 26 (process 28214):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 25 (process 28213):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 24 (process 28212):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 23 (process 28211):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 22 (process 28210):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 21 (process 28209):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 20 (process 28208):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 19 (process 28207):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 18 (process 28206):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 17 (process 28205):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 16 (process 28204):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
---Type <return> to continue, or q <return> to quit---
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 15 (process 28203):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 14 (process 28202):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
---Type <return> to continue, or q <return> to quit---
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 13 (process 28201):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 12 (process 28200):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 11 (process 28199):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 10 (process 28198):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 9 (process 28197):
---Type <return> to continue, or q <return> to quit---
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 8 (process 28196):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 7 (process 28195):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 6 (process 28194):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 5 (process 28193):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
---Type <return> to continue, or q <return> to quit---
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 4 (process 28192):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 3 (process 28191):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000000004e583f in async_main (arg=<value optimized out>)
    at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 2 (process 28190):
#0  0x0000003d05a0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x000000000046b7bc in sys_msg_dispatcher_func (
    unused=<value optimized out>) at beam/erl_threads.h:634
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6

Thread 1 (process 28189):
#0  0x0000003d05a0d2cb in read () from /lib64/libpthread.so.0
#1  0x0000000000523f6d in signal_dispatcher_thread_func (
    unused=<value optimized out>) at sys/unix/sys.c:2913
#2  0x000000000057ca34 in thr_wrapper (vtwd=<value optimized out>)
    at common/ethread.c:475
#3  0x0000003d05a06367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003d04ed309d in clone () from /lib64/libc.so.6
(gdb) 
(gdb) 
(gdb) quit

Abridged output from "cat /proc/cpuinfo":

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping        : 5
cpu MHz         : 1596.000
cache size      : 8192 KB
physical id     : 1
siblings        : 8
core id         : 0
cpu cores       : 4
[...]



More information about the erlang-bugs mailing list