[erlang-bugs] VM locks up on write to socket (and now it seems to file too)

Jonas Falkevik jonas.falkevik@REDACTED
Mon Feb 10 18:33:47 CET 2014


Hi Patrik & Peter,
do you know if this issue has been fixed in any later release of RHEL?
Any ticket id that could be shared?

Thanks,
Jonas

On Dec 6, 2012, at 13:01 , Patrik Nyblom wrote:

> Hi!
> 
> Good! The workaround isn't all that costly either, so I think we could put that in R16 as is. It would be good if the OS bug got fixed though, but I think that's also on it's way.
> 
> Cheers,
> /Patrik
> 
> On 12/05/2012 02:20 PM, Peter Membrey wrote:
>> Hi Patrik,
>> 
>> Really sorry for the delay in getting back to you!
>> 
>> I tried the same test on RHEL 6.3 using the patched version and
>> everything seems fine. No stuck threads and the VM is still happy and
>> responsive.
>> 
>> I'm currently working on a load testing app to try and trigger the
>> issue on demand in the application itself, but I suspect your patch
>> has done the trick!
>> 
>> Thanks for fixing this so fast and sorry again for the delay in
>> getting back in touch!
>> 
>> Cheers,
>> 
>> Pete
>> 
>> 
>> 
>> On 30 November 2012 02:46, Patrik Nyblom <pan@REDACTED> wrote:
>>> Hi!
>>> 
>>> On 11/29/2012 04:41 AM, Peter Membrey wrote:
>>>> Hi Patrik,
>>>> 
>>>> I can also confirm that this bug exists on Red Hat Enterprise Linux
>>>> 6.3. I'll raise a support ticket with them as well.
>>>> 
>>>> A workaround in the vm would be nice if you have time… :-)
>>> Could you try the attached diff and see if it works for your environment? It
>>> would seem nothing is written when 0 is returned, so it should be safe to
>>> try again...
>>> 
>>> Cheers,
>>> /Patrik
>>> 
>>> 
>>>> Cheers,
>>>> 
>>>> Pete
>>>> 
>>>> 
>>>> On 29 November 2012 01:23, Patrik Nyblom <pan@REDACTED> wrote:
>>>>> Hi again!
>>>>> 
>>>>> No problem reproducing when I've got CentOS 6.3... The following commands
>>>>> in
>>>>> the Erlang shell:
>>>>> {ok,L} = gen_tcp:listen(4747,[{active,false}]).
>>>>> {ok,S} = gen_tcp:connect("localhost",4747,[{active,false}]).
>>>>> {ok,A} = gen_tcp:accept(L).
>>>>> gen_tcp:send(A,binary:copy(<<$a:8>>,2158022464)).
>>>>> 
>>>>> gives the following strace:
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [.....]
>>>>> 
>>>>> While on ubuntu for example it works like it should...Looks like a kernel
>>>>> bug to me... I wonder if this should be worked around or just reported...
>>>>> I
>>>>> suppose both... Sigh...
>>>>> 
>>>>> /Patrik
>>>>> 
>>>>> 
>>>>> On 11/28/2012 05:23 PM, Peter Membrey wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> No problem, I'll do what I can to help - thanks for looking into this
>>>>>> so quickly!
>>>>>> 
>>>>>> Any idea what might be causing it?
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Pete
>>>>>> 
>>>>>> On 28 November 2012 23:54, Patrik Nyblom <pan@REDACTED> wrote:
>>>>>>> Hi!
>>>>>>> 
>>>>>>> I'll upgrade the CentOS VM I have to 6.3 (only had 6.1 :() and see if I
>>>>>>> can
>>>>>>> reproduce. If that fails, could you run a VM with a patch to try to
>>>>>>> handle
>>>>>>> the unexpected case and see if that fixes it?
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> /Patrik
>>>>>>> 
>>>>>>> On 11/24/2012 02:57 PM, Peter Membrey wrote:
>>>>>>>> Hi guys,
>>>>>>>> 
>>>>>>>> Thanks for getting back in touch so quickly!
>>>>>>>> 
>>>>>>>> I did do an lsof on the process and I can confirm that it was
>>>>>>>> definitely a socket. However by that time the application it had been
>>>>>>>> trying to send to had been killed. When I checked the sockets were
>>>>>>>> showing as waiting to close. Unfortunately I didn't think to do an
>>>>>>>> lsof until after the apps had been shut down. I was hoping the VM
>>>>>>>> would recover if I killed the app that had upset it. However even
>>>>>>>> after all the apps connected had been shut down, the issue didn't
>>>>>>>> resolve.
>>>>>>>> 
>>>>>>>> The application receives requests from a client, which contains two
>>>>>>>> data items. The stream ID and a timestamp. Both are encoded as big
>>>>>>>> integer unsigned numbers. The server then looks through the file
>>>>>>>> referenced by the stream ID and uses the timestamp as an index. The
>>>>>>>> file format is currently really simple, in the form of:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> <Timestamp:64/native-integer,Length:32/native-integer,Data:Length/binary>>
>>>>>>>> 
>>>>>>>> There is an index file that provides an offset into the file based on
>>>>>>>> time stamp, but basically it opens the file, and reads sequentially
>>>>>>>> through it until it finds the timestamps that it cares about. In this
>>>>>>>> case it reads all data with a greater timestamp until the end of the
>>>>>>>> file is reached. It's possible the client is sending an incorrect
>>>>>>>> timestamp, and maybe too much data is being read. However the loop is
>>>>>>>> very primitive - it reads all the data in one go before passing it
>>>>>>>> back to the protocol handler to send down the socket; so by that time
>>>>>>>> even though the response is technically incorrect and the app has
>>>>>>>> failed, it should still not cause the VM any issues.
>>>>>>>> 
>>>>>>>> The data is polled every 10 seconds by the client app so I would not
>>>>>>>> expect there to be 2GB of new data to send. I'm afraid my C skills are
>>>>>>>> somewhat limited, so I'm not sure how to put together a sample app to
>>>>>>>> try out writev. The platform is 64bit CentOS 6.3 (equivalent to RHEL
>>>>>>>> 6.3) so I'm not expecting any strange or weird behaviour from the OS
>>>>>>>> level but of course I could be completely wrong there. The OS is
>>>>>>>> running directly on hardware, so there's no VM layer to worry about.
>>>>>>>> 
>>>>>>>> Hope this might offer some additional clues…
>>>>>>>> 
>>>>>>>> Thanks again!
>>>>>>>> 
>>>>>>>> Kind Regards,
>>>>>>>> 
>>>>>>>> Peter Membrey
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 24 November 2012 00:13, Patrik Nyblom <pan@REDACTED> wrote:
>>>>>>>>> Hi again!
>>>>>>>>> 
>>>>>>>>> Could you go back to the version without the printouts and get back
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> situation where writev loops returning 0 (as in the strace)? If so,
>>>>>>>>> it
>>>>>>>>> would
>>>>>>>>> be really interesting to see an 'lsof' of the beam process, to see if
>>>>>>>>> this
>>>>>>>>> file descriptor really is open and is a socket...
>>>>>>>>> 
>>>>>>>>> The thing is that writev with a vector that is not empty, would never
>>>>>>>>> return
>>>>>>>>> 0 for a non blocking socket. Not on any modern (i.e. not ancient)
>>>>>>>>> POSIX
>>>>>>>>> compliant system anyway. Of course it is a *really* large item you
>>>>>>>>> are
>>>>>>>>> trying to write there, but it should be no problem for a 64bit linux.
>>>>>>>>> 
>>>>>>>>> Also I think there is no use finding the Erlang code, I'll take that
>>>>>>>>> back,
>>>>>>>>> It would be more interesting to see what really happens at the OS/VM
>>>>>>>>> level
>>>>>>>>> in this case.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Patrik
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 11/23/2012 01:49 AM, Loïc Hoguin wrote:
>>>>>>>>>> Sending this on behalf of someone who didn't manage to get the email
>>>>>>>>>> sent
>>>>>>>>>> to this list after 2 attempts. If someone can check if he's hold up
>>>>>>>>>> or
>>>>>>>>>> something that'd be great.
>>>>>>>>>> 
>>>>>>>>>> Anyway he has a big issue so I hope I can relay the conversation
>>>>>>>>>> reliably.
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>>> 
>>>>>>>>>> On 11/23/2012 01:45 AM, Peter Membrey wrote:
>>>>>>>>>>> From: Peter Membrey <peter@REDACTED>
>>>>>>>>>>> Date: 22 November 2012 19:02
>>>>>>>>>>> Subject: VM locks up on write to socket (and now it seems to file
>>>>>>>>>>> too)
>>>>>>>>>>> To: erlang-bugs@REDACTED
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hi guys,
>>>>>>>>>>> 
>>>>>>>>>>> I wrote a simple database application called CakeDB
>>>>>>>>>>> (https://github.com/pmembrey/cakedb) that basically spends its time
>>>>>>>>>>> reading and writing files and sockets. There's very little in the
>>>>>>>>>>> way
>>>>>>>>>>> of complex logic. It is running on CentOS 6.3 with all the updates
>>>>>>>>>>> applied. I hit this problem on R15B02 so I rolled back to R15B01
>>>>>>>>>>> but
>>>>>>>>>>> the issue remained. Erlang was built from source.
>>>>>>>>>>> 
>>>>>>>>>>> The machine has two Intel X5690 CPUs giving 12 cores plus HT. I've
>>>>>>>>>>> tried various arguments for the VM but so far nothing has prevented
>>>>>>>>>>> the problem. At the moment I'm using:
>>>>>>>>>>> 
>>>>>>>>>>> +K
>>>>>>>>>>> +A 6
>>>>>>>>>>> +sbt tnnps
>>>>>>>>>>> 
>>>>>>>>>>> The issue I'm seeing is that one of the scheduler threads will hit
>>>>>>>>>>> 100% cpu usage and the entire VM will become unresponsive. When
>>>>>>>>>>> this
>>>>>>>>>>> happens, I am not able to connect via the console with attach and
>>>>>>>>>>> entop is also unable to connect. I can still establish TCP
>>>>>>>>>>> connections
>>>>>>>>>>> to the application, but I never receive a response. A standard kill
>>>>>>>>>>> signal will cause the VM to shut down (it doesn't need -9).
>>>>>>>>>>> 
>>>>>>>>>>> Due to the pedigree of the VM I am quite willing to accept that
>>>>>>>>>>> I've
>>>>>>>>>>> made a fundamental mistake in my code. I am pretty sure that the
>>>>>>>>>>> way
>>>>>>>>>>> I
>>>>>>>>>>> am doing the file IO could result in some race conditions. However,
>>>>>>>>>>> my
>>>>>>>>>>> poor code aside, from what I understand, I still shouldn't be able
>>>>>>>>>>> to
>>>>>>>>>>> crash / deadlock the VM like this.
>>>>>>>>>>> 
>>>>>>>>>>> The issue doesn't seem to be caused by load. The app can fail when
>>>>>>>>>>> it's very busy, but also when it is practically idle. I haven't
>>>>>>>>>>> been
>>>>>>>>>>> able to find a trigger or any other explanation for the failure.
>>>>>>>>>>> 
>>>>>>>>>>> The thread maxing out the CPU is attempting to write data to the
>>>>>>>>>>> socket:
>>>>>>>>>>> 
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  0x00007f9882ab6377 in writev () from /lib64/libc.so.6
>>>>>>>>>>> #1  0x000000000058a81f in tcp_inet_output (data=0x2407570,
>>>>>>>>>>> event=<value optimized out>) at drivers/common/inet_drv.c:9681
>>>>>>>>>>> #2  tcp_inet_drv_output (data=0x2407570, event=<value optimized
>>>>>>>>>>> out>)
>>>>>>>>>>> at drivers/common/inet_drv.c:9601
>>>>>>>>>>> #3  0x00000000004b773f in erts_port_task_execute
>>>>>>>>>>> (runq=0x7f98826019c0,
>>>>>>>>>>> curr_port_pp=0x7f9881639338)  at beam/erl_port_task.c:858
>>>>>>>>>>> #4  0x00000000004afd83 in schedule (p=<value optimized out>,
>>>>>>>>>>> calls=<value optimized out>) at beam/erl_process.c:6533
>>>>>>>>>>> #5  0x0000000000539ca2 in process_main () at beam/beam_emu.c:1268
>>>>>>>>>>> #6  0x00000000004b1279 in sched_thread_func (vesdp=0x7f9881639280)
>>>>>>>>>>> at
>>>>>>>>>>> beam/erl_process.c:4834
>>>>>>>>>>> #7  0x00000000005ba726 in thr_wrapper (vtwd=0x7fff6cfe2300) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #8  0x00007f9882f78851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #9  0x00007f9882abe11d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>> 
>>>>>>>>>>> I then tried running strace on that thread and got (indefinitely):
>>>>>>>>>>> 
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1)       = 0
>>>>>>>>>>> ...
>>>>>>>>>>> 
>>>>>>>>>>>     From what I can tell, it's trying to write data to a socket,
>>>>>>>>>>> which
>>>>>>>>>>> is
>>>>>>>>>>> succeeding, but writing 0 bytes. From the earlier definitions in
>>>>>>>>>>> the
>>>>>>>>>>> source file, an error condition would be signified by a negative
>>>>>>>>>>> number. Any other result is the number of bytes written, in this
>>>>>>>>>>> case
>>>>>>>>>>> 0. I'm not sure if this is desired behaviour or not. I've tried
>>>>>>>>>>> killing the application on the other end of the socket, but it has
>>>>>>>>>>> no
>>>>>>>>>>> effect on the VM.
>>>>>>>>>>> 
>>>>>>>>>>> I have enabled debugging for the inet code, so hopefully this will
>>>>>>>>>>> give a little more insight. I am currently trying to reproduce the
>>>>>>>>>>> condition, but as I really have no idea what causes it, it's pretty
>>>>>>>>>>> much a case of wait and see.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> **** UPDATE ****
>>>>>>>>>>> 
>>>>>>>>>>> I managed to lock up the VM again, but this time it was caused by
>>>>>>>>>>> file
>>>>>>>>>>> IO,
>>>>>>>>>>> probably from the debugging statements. Although it worked fine for
>>>>>>>>>>> some
>>>>>>>>>>> time
>>>>>>>>>>> the last entry in the file was cut off.
>>>>>>>>>>> 
>>>>>>>>>>>     From GDB:
>>>>>>>>>>> 
>>>>>>>>>>> (gdb) info threads
>>>>>>>>>>>       53 Thread 0x7f83e988b700 (LWP 8621)  0x00007f83ea6da54d in
>>>>>>>>>>> read
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libpthread.so.0
>>>>>>>>>>>       52 Thread 0x7f83e8c8f700 (LWP 8622)  0x00007f83ea6d743c in
>>>>>>>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>>>>>>>>>>       51 Thread 0x7f83e818d700 (LWP 8623)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       50 Thread 0x7f83e816b700 (LWP 8624)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       49 Thread 0x7f83e8149700 (LWP 8625)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       48 Thread 0x7f83e8127700 (LWP 8626)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       47 Thread 0x7f83e8105700 (LWP 8627)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       46 Thread 0x7f83e80e3700 (LWP 8628)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       45 Thread 0x7f83e80c1700 (LWP 8629)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       44 Thread 0x7f83e809f700 (LWP 8630)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       43 Thread 0x7f83e807d700 (LWP 8631)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       42 Thread 0x7f83e805b700 (LWP 8632)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       41 Thread 0x7f83e8039700 (LWP 8633)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       40 Thread 0x7f83e8017700 (LWP 8634)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       39 Thread 0x7f83e7ff5700 (LWP 8635)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       38 Thread 0x7f83e7fd3700 (LWP 8636)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       37 Thread 0x7f83e7fb1700 (LWP 8637)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       36 Thread 0x7f83e7f8f700 (LWP 8638)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       35 Thread 0x7f83e7f6d700 (LWP 8639)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       34 Thread 0x7f83e7f4b700 (LWP 8640)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       33 Thread 0x7f83e7f29700 (LWP 8641)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       32 Thread 0x7f83e7f07700 (LWP 8642)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       31 Thread 0x7f83e7ee5700 (LWP 8643)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       30 Thread 0x7f83e7ec3700 (LWP 8644)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       29 Thread 0x7f83e7ea1700 (LWP 8645)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       28 Thread 0x7f83e7e7f700 (LWP 8646)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       27 Thread 0x7f83d7c5a700 (LWP 8647)  0x00007f83ea6db09d in
>>>>>>>>>>> waitpid
>>>>>>>>>>> () from /lib64/libpthread.so.0
>>>>>>>>>>>       26 Thread 0x7f83d7c53700 (LWP 8648)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       25 Thread 0x7f83d7252700 (LWP 8649)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       24 Thread 0x7f83d6851700 (LWP 8650)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       23 Thread 0x7f83d5e50700 (LWP 8651)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       22 Thread 0x7f83d544f700 (LWP 8652)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       21 Thread 0x7f83d4a4e700 (LWP 8653)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       20 Thread 0x7f83d404d700 (LWP 8654)  0x00007f83ea20be7d in
>>>>>>>>>>> write
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       19 Thread 0x7f83d364c700 (LWP 8655)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       18 Thread 0x7f83d2c4b700 (LWP 8656)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       17 Thread 0x7f83d224a700 (LWP 8657)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       16 Thread 0x7f83d1849700 (LWP 8658)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       15 Thread 0x7f83d0e48700 (LWP 8659)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       14 Thread 0x7f83d0447700 (LWP 8660)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       13 Thread 0x7f83cfa46700 (LWP 8661)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       12 Thread 0x7f83cf045700 (LWP 8662)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       11 Thread 0x7f83ce644700 (LWP 8663)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       10 Thread 0x7f83cdc43700 (LWP 8664)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>>       9 Thread 0x7f83cd242700 (LWP 8665)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       8 Thread 0x7f83cc841700 (LWP 8666)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       7 Thread 0x7f83cbe40700 (LWP 8667)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       6 Thread 0x7f83cb43f700 (LWP 8668)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       5 Thread 0x7f83caa3e700 (LWP 8669)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       4 Thread 0x7f83ca03d700 (LWP 8670)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       3 Thread 0x7f83c963c700 (LWP 8671)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>>       2 Thread 0x7f83c8c3b700 (LWP 8672)  0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> * 1 Thread 0x7f83eb3a8700 (LWP 8597)  0x00007f83ea211d03 in select
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  0x00007f83ea20be7d in write () from /lib64/libc.so.6
>>>>>>>>>>> #1  0x00007f83ea1a2583 in _IO_new_file_write () from
>>>>>>>>>>> /lib64/libc.so.6
>>>>>>>>>>> #2  0x00007f83ea1a3b35 in _IO_new_do_write () from /lib64/libc.so.6
>>>>>>>>>>> #3  0x00007f83ea1a21fd in _IO_new_file_xsputn () from
>>>>>>>>>>> /lib64/libc.so.6
>>>>>>>>>>> #4  0x00007f83ea17589d in vfprintf () from /lib64/libc.so.6
>>>>>>>>>>> #5  0x00007f83ea18003a in printf () from /lib64/libc.so.6
>>>>>>>>>>> #6  0x000000000058f0e8 in tcp_recv (desc=0x2c3d350, request_len=0)
>>>>>>>>>>> at
>>>>>>>>>>> drivers/common/inet_drv.c:8976
>>>>>>>>>>> #7  0x000000000058f63a in tcp_inet_input (data=0x2c3d350,
>>>>>>>>>>> event=<value
>>>>>>>>>>> optimized out>) at drivers/common/inet_drv.c:9326
>>>>>>>>>>> #8  tcp_inet_drv_input (data=0x2c3d350, event=<value optimized
>>>>>>>>>>> out>)
>>>>>>>>>>> at drivers/common/inet_drv.c:9604
>>>>>>>>>>> #9  0x00000000004b770f in erts_port_task_execute
>>>>>>>>>>> (runq=0x7f83e9d5d3c0,
>>>>>>>>>>> curr_port_pp=0x7f83e8dc6e78) at beam/erl_port_task.c:851
>>>>>>>>>>> #10 0x00000000004afd83 in schedule (p=<value optimized out>,
>>>>>>>>>>> calls=<value optimized out>) at beam/erl_process.c:6533
>>>>>>>>>>> #11 0x0000000000539ca2 in process_main () at beam/beam_emu.c:1268
>>>>>>>>>>> #12 0x00000000004b1279 in sched_thread_func (vesdp=0x7f83e8dc6dc0)
>>>>>>>>>>> at
>>>>>>>>>>> beam/erl_process.c:4834
>>>>>>>>>>> #13 0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266da0) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #14 0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #15 0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>> 
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  0x00007f83ea6da54d in read () from /lib64/libpthread.so.0
>>>>>>>>>>> #1  0x0000000000554b6e in signal_dispatcher_thread_func
>>>>>>>>>>> (unused=<value
>>>>>>>>>>> optimized out>) at sys/unix/sys.c:2776
>>>>>>>>>>> #2  0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266c80) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #3  0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #4  0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>> 
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  0x00007f83ea215ae9 in syscall () from /lib64/libc.so.6
>>>>>>>>>>> #1  0x00000000005bba35 in wait__ (e=0x2989390) at
>>>>>>>>>>> pthread/ethr_event.c:92
>>>>>>>>>>> #2  ethr_event_wait (e=0x2989390) at pthread/ethr_event.c:218
>>>>>>>>>>> #3  0x00000000004ae5bd in erts_tse_wait (fcalls=<value optimized
>>>>>>>>>>> out>,
>>>>>>>>>>> esdp=0x7f83e8e2c440, rq=0x7f83e9d5e7c0) at beam/erl_threads.h:2319
>>>>>>>>>>> #4  scheduler_wait (fcalls=<value optimized out>,
>>>>>>>>>>> esdp=0x7f83e8e2c440,
>>>>>>>>>>> rq=0x7f83e9d5e7c0) at beam/erl_process.c:2087
>>>>>>>>>>> #5  0x00000000004afb94 in schedule (p=<value optimized out>,
>>>>>>>>>>> calls=<value optimized out>) at beam/erl_process.c:6467
>>>>>>>>>>> #6  0x0000000000539ca2 in process_main () at beam/beam_emu.c:1268
>>>>>>>>>>> #7  0x00000000004b1279 in sched_thread_func (vesdp=0x7f83e8e2c440)
>>>>>>>>>>> at
>>>>>>>>>>> beam/erl_process.c:4834
>>>>>>>>>>> #8  0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266da0) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #9  0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #10 0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0  0x00007f83ea6db09d in waitpid () from /lib64/libpthread.so.0
>>>>>>>>>>> #1  0x0000000000555a9f in child_waiter (unused=<value optimized
>>>>>>>>>>> out>)
>>>>>>>>>>> at sys/unix/sys.c:2700
>>>>>>>>>>> #2  0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266d50) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #3  0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #4  0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> **** END UPDATE ****
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I'm happy to provide any information I can, so please don't
>>>>>>>>>>> hesitate
>>>>>>>>>>> to
>>>>>>>>>>> ask.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks in advance!
>>>>>>>>>>> 
>>>>>>>>>>> Kind Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Peter Membrey
>>>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> erlang-bugs mailing list
>>>>>>>>> erlang-bugs@REDACTED
>>>>>>>>> http://erlang.org/mailman/listinfo/erlang-bugs
>>>>>>> 
> 
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://erlang.org/mailman/listinfo/erlang-bugs




More information about the erlang-bugs mailing list