[erlang-bugs] VM locks up on write to socket (and now it seems to file too)
Peter Membrey
peter@REDACTED
Thu Jan 17 11:43:58 CET 2013
Hi all,
Found a weird issue that might be related to this patch, but I'm not
entirely sure. Basically my application writes data in append mode
with delayed_write to a file. The data is only written by one process
although it can be read directly off disk from multiple other
processes. I've found recently that some messages appear to be
truncated. Their structure looks something like this:
<3 byte header><payload<3 byte footer>
The strange thing is the header contains a CRC32 check and size that
matches what the payload should be. The footer is added at the same
time as the header. So in theory, if you have the header you should
have the footer. Yet what I'm seeing is the header plus only a small
part of the message itself, followed by a new header. I can't see
anything in the code that would cause it to do this. Literally it puts
messages into a list and then every now and then flattens the list and
writes it out...
I'm still trying to verify it, but it's possible that the corrupt
message is written again in full later. I'm seeing data with a
duplicate ID ( which should be unique ) but I have to confirm this.
Anyway I thought I'd though this out there just in case it's related
to the kernel bug as this issue doesn't seem to be in the data I
collected before the fix...
Sorry for the vagueness...
Kind Regards,
Peter Membrey
On 6 December 2012 20:01, Patrik Nyblom <pan@REDACTED> wrote:
> Hi!
>
> Good! The workaround isn't all that costly either, so I think we could put
> that in R16 as is. It would be good if the OS bug got fixed though, but I
> think that's also on it's way.
>
> Cheers,
> /Patrik
>
>
> On 12/05/2012 02:20 PM, Peter Membrey wrote:
>>
>> Hi Patrik,
>>
>> Really sorry for the delay in getting back to you!
>>
>> I tried the same test on RHEL 6.3 using the patched version and
>> everything seems fine. No stuck threads and the VM is still happy and
>> responsive.
>>
>> I'm currently working on a load testing app to try and trigger the
>> issue on demand in the application itself, but I suspect your patch
>> has done the trick!
>>
>> Thanks for fixing this so fast and sorry again for the delay in
>> getting back in touch!
>>
>> Cheers,
>>
>> Pete
>>
>>
>>
>> On 30 November 2012 02:46, Patrik Nyblom <pan@REDACTED> wrote:
>>>
>>> Hi!
>>>
>>> On 11/29/2012 04:41 AM, Peter Membrey wrote:
>>>>
>>>> Hi Patrik,
>>>>
>>>> I can also confirm that this bug exists on Red Hat Enterprise Linux
>>>> 6.3. I'll raise a support ticket with them as well.
>>>>
>>>> A workaround in the vm would be nice if you have time… :-)
>>>
>>> Could you try the attached diff and see if it works for your environment?
>>> It
>>> would seem nothing is written when 0 is returned, so it should be safe to
>>> try again...
>>>
>>> Cheers,
>>> /Patrik
>>>
>>>
>>>> Cheers,
>>>>
>>>> Pete
>>>>
>>>>
>>>> On 29 November 2012 01:23, Patrik Nyblom <pan@REDACTED> wrote:
>>>>>
>>>>> Hi again!
>>>>>
>>>>> No problem reproducing when I've got CentOS 6.3... The following
>>>>> commands
>>>>> in
>>>>> the Erlang shell:
>>>>> {ok,L} = gen_tcp:listen(4747,[{active,false}]).
>>>>> {ok,S} = gen_tcp:connect("localhost",4747,[{active,false}]).
>>>>> {ok,A} = gen_tcp:accept(L).
>>>>> gen_tcp:send(A,binary:copy(<<$a:8>>,2158022464)).
>>>>>
>>>>> gives the following strace:
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [pid 15859] writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"...,
>>>>> 2158022464}], 1) = 0
>>>>> [.....]
>>>>>
>>>>> While on ubuntu for example it works like it should...Looks like a
>>>>> kernel
>>>>> bug to me... I wonder if this should be worked around or just
>>>>> reported...
>>>>> I
>>>>> suppose both... Sigh...
>>>>>
>>>>> /Patrik
>>>>>
>>>>>
>>>>> On 11/28/2012 05:23 PM, Peter Membrey wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> No problem, I'll do what I can to help - thanks for looking into this
>>>>>> so quickly!
>>>>>>
>>>>>> Any idea what might be causing it?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Pete
>>>>>>
>>>>>> On 28 November 2012 23:54, Patrik Nyblom <pan@REDACTED> wrote:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I'll upgrade the CentOS VM I have to 6.3 (only had 6.1 :() and see if
>>>>>>> I
>>>>>>> can
>>>>>>> reproduce. If that fails, could you run a VM with a patch to try to
>>>>>>> handle
>>>>>>> the unexpected case and see if that fixes it?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> /Patrik
>>>>>>>
>>>>>>> On 11/24/2012 02:57 PM, Peter Membrey wrote:
>>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>> Thanks for getting back in touch so quickly!
>>>>>>>>
>>>>>>>> I did do an lsof on the process and I can confirm that it was
>>>>>>>> definitely a socket. However by that time the application it had
>>>>>>>> been
>>>>>>>> trying to send to had been killed. When I checked the sockets were
>>>>>>>> showing as waiting to close. Unfortunately I didn't think to do an
>>>>>>>> lsof until after the apps had been shut down. I was hoping the VM
>>>>>>>> would recover if I killed the app that had upset it. However even
>>>>>>>> after all the apps connected had been shut down, the issue didn't
>>>>>>>> resolve.
>>>>>>>>
>>>>>>>> The application receives requests from a client, which contains two
>>>>>>>> data items. The stream ID and a timestamp. Both are encoded as big
>>>>>>>> integer unsigned numbers. The server then looks through the file
>>>>>>>> referenced by the stream ID and uses the timestamp as an index. The
>>>>>>>> file format is currently really simple, in the form of:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> <Timestamp:64/native-integer,Length:32/native-integer,Data:Length/binary>>
>>>>>>>>
>>>>>>>> There is an index file that provides an offset into the file based
>>>>>>>> on
>>>>>>>> time stamp, but basically it opens the file, and reads sequentially
>>>>>>>> through it until it finds the timestamps that it cares about. In
>>>>>>>> this
>>>>>>>> case it reads all data with a greater timestamp until the end of the
>>>>>>>> file is reached. It's possible the client is sending an incorrect
>>>>>>>> timestamp, and maybe too much data is being read. However the loop
>>>>>>>> is
>>>>>>>> very primitive - it reads all the data in one go before passing it
>>>>>>>> back to the protocol handler to send down the socket; so by that
>>>>>>>> time
>>>>>>>> even though the response is technically incorrect and the app has
>>>>>>>> failed, it should still not cause the VM any issues.
>>>>>>>>
>>>>>>>> The data is polled every 10 seconds by the client app so I would not
>>>>>>>> expect there to be 2GB of new data to send. I'm afraid my C skills
>>>>>>>> are
>>>>>>>> somewhat limited, so I'm not sure how to put together a sample app
>>>>>>>> to
>>>>>>>> try out writev. The platform is 64bit CentOS 6.3 (equivalent to RHEL
>>>>>>>> 6.3) so I'm not expecting any strange or weird behaviour from the OS
>>>>>>>> level but of course I could be completely wrong there. The OS is
>>>>>>>> running directly on hardware, so there's no VM layer to worry about.
>>>>>>>>
>>>>>>>> Hope this might offer some additional clues…
>>>>>>>>
>>>>>>>> Thanks again!
>>>>>>>>
>>>>>>>> Kind Regards,
>>>>>>>>
>>>>>>>> Peter Membrey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 24 November 2012 00:13, Patrik Nyblom <pan@REDACTED> wrote:
>>>>>>>>>
>>>>>>>>> Hi again!
>>>>>>>>>
>>>>>>>>> Could you go back to the version without the printouts and get back
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> situation where writev loops returning 0 (as in the strace)? If so,
>>>>>>>>> it
>>>>>>>>> would
>>>>>>>>> be really interesting to see an 'lsof' of the beam process, to see
>>>>>>>>> if
>>>>>>>>> this
>>>>>>>>> file descriptor really is open and is a socket...
>>>>>>>>>
>>>>>>>>> The thing is that writev with a vector that is not empty, would
>>>>>>>>> never
>>>>>>>>> return
>>>>>>>>> 0 for a non blocking socket. Not on any modern (i.e. not ancient)
>>>>>>>>> POSIX
>>>>>>>>> compliant system anyway. Of course it is a *really* large item you
>>>>>>>>> are
>>>>>>>>> trying to write there, but it should be no problem for a 64bit
>>>>>>>>> linux.
>>>>>>>>>
>>>>>>>>> Also I think there is no use finding the Erlang code, I'll take
>>>>>>>>> that
>>>>>>>>> back,
>>>>>>>>> It would be more interesting to see what really happens at the
>>>>>>>>> OS/VM
>>>>>>>>> level
>>>>>>>>> in this case.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Patrik
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 11/23/2012 01:49 AM, Loïc Hoguin wrote:
>>>>>>>>>>
>>>>>>>>>> Sending this on behalf of someone who didn't manage to get the
>>>>>>>>>> email
>>>>>>>>>> sent
>>>>>>>>>> to this list after 2 attempts. If someone can check if he's hold
>>>>>>>>>> up
>>>>>>>>>> or
>>>>>>>>>> something that'd be great.
>>>>>>>>>>
>>>>>>>>>> Anyway he has a big issue so I hope I can relay the conversation
>>>>>>>>>> reliably.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> On 11/23/2012 01:45 AM, Peter Membrey wrote:
>>>>>>>>>>>
>>>>>>>>>>> From: Peter Membrey <peter@REDACTED>
>>>>>>>>>>> Date: 22 November 2012 19:02
>>>>>>>>>>> Subject: VM locks up on write to socket (and now it seems to file
>>>>>>>>>>> too)
>>>>>>>>>>> To: erlang-bugs@REDACTED
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>
>>>>>>>>>>> I wrote a simple database application called CakeDB
>>>>>>>>>>> (https://github.com/pmembrey/cakedb) that basically spends its
>>>>>>>>>>> time
>>>>>>>>>>> reading and writing files and sockets. There's very little in the
>>>>>>>>>>> way
>>>>>>>>>>> of complex logic. It is running on CentOS 6.3 with all the
>>>>>>>>>>> updates
>>>>>>>>>>> applied. I hit this problem on R15B02 so I rolled back to R15B01
>>>>>>>>>>> but
>>>>>>>>>>> the issue remained. Erlang was built from source.
>>>>>>>>>>>
>>>>>>>>>>> The machine has two Intel X5690 CPUs giving 12 cores plus HT.
>>>>>>>>>>> I've
>>>>>>>>>>> tried various arguments for the VM but so far nothing has
>>>>>>>>>>> prevented
>>>>>>>>>>> the problem. At the moment I'm using:
>>>>>>>>>>>
>>>>>>>>>>> +K
>>>>>>>>>>> +A 6
>>>>>>>>>>> +sbt tnnps
>>>>>>>>>>>
>>>>>>>>>>> The issue I'm seeing is that one of the scheduler threads will
>>>>>>>>>>> hit
>>>>>>>>>>> 100% cpu usage and the entire VM will become unresponsive. When
>>>>>>>>>>> this
>>>>>>>>>>> happens, I am not able to connect via the console with attach and
>>>>>>>>>>> entop is also unable to connect. I can still establish TCP
>>>>>>>>>>> connections
>>>>>>>>>>> to the application, but I never receive a response. A standard
>>>>>>>>>>> kill
>>>>>>>>>>> signal will cause the VM to shut down (it doesn't need -9).
>>>>>>>>>>>
>>>>>>>>>>> Due to the pedigree of the VM I am quite willing to accept that
>>>>>>>>>>> I've
>>>>>>>>>>> made a fundamental mistake in my code. I am pretty sure that the
>>>>>>>>>>> way
>>>>>>>>>>> I
>>>>>>>>>>> am doing the file IO could result in some race conditions.
>>>>>>>>>>> However,
>>>>>>>>>>> my
>>>>>>>>>>> poor code aside, from what I understand, I still shouldn't be
>>>>>>>>>>> able
>>>>>>>>>>> to
>>>>>>>>>>> crash / deadlock the VM like this.
>>>>>>>>>>>
>>>>>>>>>>> The issue doesn't seem to be caused by load. The app can fail
>>>>>>>>>>> when
>>>>>>>>>>> it's very busy, but also when it is practically idle. I haven't
>>>>>>>>>>> been
>>>>>>>>>>> able to find a trigger or any other explanation for the failure.
>>>>>>>>>>>
>>>>>>>>>>> The thread maxing out the CPU is attempting to write data to the
>>>>>>>>>>> socket:
>>>>>>>>>>>
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0 0x00007f9882ab6377 in writev () from /lib64/libc.so.6
>>>>>>>>>>> #1 0x000000000058a81f in tcp_inet_output (data=0x2407570,
>>>>>>>>>>> event=<value optimized out>) at drivers/common/inet_drv.c:9681
>>>>>>>>>>> #2 tcp_inet_drv_output (data=0x2407570, event=<value optimized
>>>>>>>>>>> out>)
>>>>>>>>>>> at drivers/common/inet_drv.c:9601
>>>>>>>>>>> #3 0x00000000004b773f in erts_port_task_execute
>>>>>>>>>>> (runq=0x7f98826019c0,
>>>>>>>>>>> curr_port_pp=0x7f9881639338) at beam/erl_port_task.c:858
>>>>>>>>>>> #4 0x00000000004afd83 in schedule (p=<value optimized out>,
>>>>>>>>>>> calls=<value optimized out>) at beam/erl_process.c:6533
>>>>>>>>>>> #5 0x0000000000539ca2 in process_main () at beam/beam_emu.c:1268
>>>>>>>>>>> #6 0x00000000004b1279 in sched_thread_func
>>>>>>>>>>> (vesdp=0x7f9881639280)
>>>>>>>>>>> at
>>>>>>>>>>> beam/erl_process.c:4834
>>>>>>>>>>> #7 0x00000000005ba726 in thr_wrapper (vtwd=0x7fff6cfe2300) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #8 0x00007f9882f78851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #9 0x00007f9882abe11d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>>
>>>>>>>>>>> I then tried running strace on that thread and got
>>>>>>>>>>> (indefinitely):
>>>>>>>>>>>
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> writev(15, [{"", 2158022464}], 1) = 0
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>> From what I can tell, it's trying to write data to a socket,
>>>>>>>>>>> which
>>>>>>>>>>> is
>>>>>>>>>>> succeeding, but writing 0 bytes. From the earlier definitions in
>>>>>>>>>>> the
>>>>>>>>>>> source file, an error condition would be signified by a negative
>>>>>>>>>>> number. Any other result is the number of bytes written, in this
>>>>>>>>>>> case
>>>>>>>>>>> 0. I'm not sure if this is desired behaviour or not. I've tried
>>>>>>>>>>> killing the application on the other end of the socket, but it
>>>>>>>>>>> has
>>>>>>>>>>> no
>>>>>>>>>>> effect on the VM.
>>>>>>>>>>>
>>>>>>>>>>> I have enabled debugging for the inet code, so hopefully this
>>>>>>>>>>> will
>>>>>>>>>>> give a little more insight. I am currently trying to reproduce
>>>>>>>>>>> the
>>>>>>>>>>> condition, but as I really have no idea what causes it, it's
>>>>>>>>>>> pretty
>>>>>>>>>>> much a case of wait and see.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> **** UPDATE ****
>>>>>>>>>>>
>>>>>>>>>>> I managed to lock up the VM again, but this time it was caused by
>>>>>>>>>>> file
>>>>>>>>>>> IO,
>>>>>>>>>>> probably from the debugging statements. Although it worked fine
>>>>>>>>>>> for
>>>>>>>>>>> some
>>>>>>>>>>> time
>>>>>>>>>>> the last entry in the file was cut off.
>>>>>>>>>>>
>>>>>>>>>>> From GDB:
>>>>>>>>>>>
>>>>>>>>>>> (gdb) info threads
>>>>>>>>>>> 53 Thread 0x7f83e988b700 (LWP 8621) 0x00007f83ea6da54d in
>>>>>>>>>>> read
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libpthread.so.0
>>>>>>>>>>> 52 Thread 0x7f83e8c8f700 (LWP 8622) 0x00007f83ea6d743c in
>>>>>>>>>>> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>>>>>>>>>> 51 Thread 0x7f83e818d700 (LWP 8623) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 50 Thread 0x7f83e816b700 (LWP 8624) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 49 Thread 0x7f83e8149700 (LWP 8625) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 48 Thread 0x7f83e8127700 (LWP 8626) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 47 Thread 0x7f83e8105700 (LWP 8627) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 46 Thread 0x7f83e80e3700 (LWP 8628) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 45 Thread 0x7f83e80c1700 (LWP 8629) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 44 Thread 0x7f83e809f700 (LWP 8630) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 43 Thread 0x7f83e807d700 (LWP 8631) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 42 Thread 0x7f83e805b700 (LWP 8632) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 41 Thread 0x7f83e8039700 (LWP 8633) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 40 Thread 0x7f83e8017700 (LWP 8634) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 39 Thread 0x7f83e7ff5700 (LWP 8635) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 38 Thread 0x7f83e7fd3700 (LWP 8636) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 37 Thread 0x7f83e7fb1700 (LWP 8637) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 36 Thread 0x7f83e7f8f700 (LWP 8638) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 35 Thread 0x7f83e7f6d700 (LWP 8639) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 34 Thread 0x7f83e7f4b700 (LWP 8640) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 33 Thread 0x7f83e7f29700 (LWP 8641) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 32 Thread 0x7f83e7f07700 (LWP 8642) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 31 Thread 0x7f83e7ee5700 (LWP 8643) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 30 Thread 0x7f83e7ec3700 (LWP 8644) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 29 Thread 0x7f83e7ea1700 (LWP 8645) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 28 Thread 0x7f83e7e7f700 (LWP 8646) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 27 Thread 0x7f83d7c5a700 (LWP 8647) 0x00007f83ea6db09d in
>>>>>>>>>>> waitpid
>>>>>>>>>>> () from /lib64/libpthread.so.0
>>>>>>>>>>> 26 Thread 0x7f83d7c53700 (LWP 8648) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 25 Thread 0x7f83d7252700 (LWP 8649) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 24 Thread 0x7f83d6851700 (LWP 8650) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 23 Thread 0x7f83d5e50700 (LWP 8651) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 22 Thread 0x7f83d544f700 (LWP 8652) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 21 Thread 0x7f83d4a4e700 (LWP 8653) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 20 Thread 0x7f83d404d700 (LWP 8654) 0x00007f83ea20be7d in
>>>>>>>>>>> write
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 19 Thread 0x7f83d364c700 (LWP 8655) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 18 Thread 0x7f83d2c4b700 (LWP 8656) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 17 Thread 0x7f83d224a700 (LWP 8657) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 16 Thread 0x7f83d1849700 (LWP 8658) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 15 Thread 0x7f83d0e48700 (LWP 8659) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 14 Thread 0x7f83d0447700 (LWP 8660) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 13 Thread 0x7f83cfa46700 (LWP 8661) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 12 Thread 0x7f83cf045700 (LWP 8662) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 11 Thread 0x7f83ce644700 (LWP 8663) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 10 Thread 0x7f83cdc43700 (LWP 8664) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> () from /lib64/libc.so.6
>>>>>>>>>>> 9 Thread 0x7f83cd242700 (LWP 8665) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 8 Thread 0x7f83cc841700 (LWP 8666) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 7 Thread 0x7f83cbe40700 (LWP 8667) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 6 Thread 0x7f83cb43f700 (LWP 8668) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 5 Thread 0x7f83caa3e700 (LWP 8669) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 4 Thread 0x7f83ca03d700 (LWP 8670) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 3 Thread 0x7f83c963c700 (LWP 8671) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> 2 Thread 0x7f83c8c3b700 (LWP 8672) 0x00007f83ea215ae9 in
>>>>>>>>>>> syscall
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> * 1 Thread 0x7f83eb3a8700 (LWP 8597) 0x00007f83ea211d03 in
>>>>>>>>>>> select
>>>>>>>>>>> ()
>>>>>>>>>>> from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0 0x00007f83ea20be7d in write () from /lib64/libc.so.6
>>>>>>>>>>> #1 0x00007f83ea1a2583 in _IO_new_file_write () from
>>>>>>>>>>> /lib64/libc.so.6
>>>>>>>>>>> #2 0x00007f83ea1a3b35 in _IO_new_do_write () from
>>>>>>>>>>> /lib64/libc.so.6
>>>>>>>>>>> #3 0x00007f83ea1a21fd in _IO_new_file_xsputn () from
>>>>>>>>>>> /lib64/libc.so.6
>>>>>>>>>>> #4 0x00007f83ea17589d in vfprintf () from /lib64/libc.so.6
>>>>>>>>>>> #5 0x00007f83ea18003a in printf () from /lib64/libc.so.6
>>>>>>>>>>> #6 0x000000000058f0e8 in tcp_recv (desc=0x2c3d350,
>>>>>>>>>>> request_len=0)
>>>>>>>>>>> at
>>>>>>>>>>> drivers/common/inet_drv.c:8976
>>>>>>>>>>> #7 0x000000000058f63a in tcp_inet_input (data=0x2c3d350,
>>>>>>>>>>> event=<value
>>>>>>>>>>> optimized out>) at drivers/common/inet_drv.c:9326
>>>>>>>>>>> #8 tcp_inet_drv_input (data=0x2c3d350, event=<value optimized
>>>>>>>>>>> out>)
>>>>>>>>>>> at drivers/common/inet_drv.c:9604
>>>>>>>>>>> #9 0x00000000004b770f in erts_port_task_execute
>>>>>>>>>>> (runq=0x7f83e9d5d3c0,
>>>>>>>>>>> curr_port_pp=0x7f83e8dc6e78) at beam/erl_port_task.c:851
>>>>>>>>>>> #10 0x00000000004afd83 in schedule (p=<value optimized out>,
>>>>>>>>>>> calls=<value optimized out>) at beam/erl_process.c:6533
>>>>>>>>>>> #11 0x0000000000539ca2 in process_main () at beam/beam_emu.c:1268
>>>>>>>>>>> #12 0x00000000004b1279 in sched_thread_func
>>>>>>>>>>> (vesdp=0x7f83e8dc6dc0)
>>>>>>>>>>> at
>>>>>>>>>>> beam/erl_process.c:4834
>>>>>>>>>>> #13 0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266da0) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #14 0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #15 0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>>
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0 0x00007f83ea6da54d in read () from /lib64/libpthread.so.0
>>>>>>>>>>> #1 0x0000000000554b6e in signal_dispatcher_thread_func
>>>>>>>>>>> (unused=<value
>>>>>>>>>>> optimized out>) at sys/unix/sys.c:2776
>>>>>>>>>>> #2 0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266c80) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #3 0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #4 0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>>
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0 0x00007f83ea215ae9 in syscall () from /lib64/libc.so.6
>>>>>>>>>>> #1 0x00000000005bba35 in wait__ (e=0x2989390) at
>>>>>>>>>>> pthread/ethr_event.c:92
>>>>>>>>>>> #2 ethr_event_wait (e=0x2989390) at pthread/ethr_event.c:218
>>>>>>>>>>> #3 0x00000000004ae5bd in erts_tse_wait (fcalls=<value optimized
>>>>>>>>>>> out>,
>>>>>>>>>>> esdp=0x7f83e8e2c440, rq=0x7f83e9d5e7c0) at
>>>>>>>>>>> beam/erl_threads.h:2319
>>>>>>>>>>> #4 scheduler_wait (fcalls=<value optimized out>,
>>>>>>>>>>> esdp=0x7f83e8e2c440,
>>>>>>>>>>> rq=0x7f83e9d5e7c0) at beam/erl_process.c:2087
>>>>>>>>>>> #5 0x00000000004afb94 in schedule (p=<value optimized out>,
>>>>>>>>>>> calls=<value optimized out>) at beam/erl_process.c:6467
>>>>>>>>>>> #6 0x0000000000539ca2 in process_main () at beam/beam_emu.c:1268
>>>>>>>>>>> #7 0x00000000004b1279 in sched_thread_func
>>>>>>>>>>> (vesdp=0x7f83e8e2c440)
>>>>>>>>>>> at
>>>>>>>>>>> beam/erl_process.c:4834
>>>>>>>>>>> #8 0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266da0) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #9 0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #10 0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0 0x00007f83ea6db09d in waitpid () from /lib64/libpthread.so.0
>>>>>>>>>>> #1 0x0000000000555a9f in child_waiter (unused=<value optimized
>>>>>>>>>>> out>)
>>>>>>>>>>> at sys/unix/sys.c:2700
>>>>>>>>>>> #2 0x00000000005bb3e6 in thr_wrapper (vtwd=0x7fffe8266d50) at
>>>>>>>>>>> pthread/ethread.c:106
>>>>>>>>>>> #3 0x00007f83ea6d3851 in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #4 0x00007f83ea21911d in clone () from /lib64/libc.so.6
>>>>>>>>>>> (gdb)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> **** END UPDATE ****
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm happy to provide any information I can, so please don't
>>>>>>>>>>> hesitate
>>>>>>>>>>> to
>>>>>>>>>>> ask.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance!
>>>>>>>>>>>
>>>>>>>>>>> Kind Regards,
>>>>>>>>>>>
>>>>>>>>>>> Peter Membrey
>>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> erlang-bugs mailing list
>>>>>>>>> erlang-bugs@REDACTED
>>>>>>>>> http://erlang.org/mailman/listinfo/erlang-bugs
>>>>>>>
>>>>>>>
>
More information about the erlang-bugs
mailing list