[erlang-questions] beam.smp[20162]: segfault at 00002aaad3e84bc8 rip 00000000004ee448 rsp 0000000042877c60 error 4

[Steven]

Good day everyone

We recently experienced a segmentation fault on one of our runtimes. The
runtime was carrying traffic and been running for about a year. Running
R13B4 64bit with smp enabled and OS is redhat 5.2 64bit. The runtime is
started with +S 8 +A 32 and not using any nifs. Unfortunately, no core dump
and this is the first time we seeing it. There's another 5 nodes with the
same deployment and hasn't happened on those runtimes (touch wood). Below
is the information I could extract. I can enable core dump but won't know
when this will happen again.

*Erlang
/usr/local/erlang/R13B-4/lib/erlang/bin/erl
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:16:16] [rq:16]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)

*beam.smp
strings -a /usr/local/erlang/R13B-4/lib/erlang/erts-5.7.5/bin/beam.smp |
fgrep GCC | more
GCC: (GNU) 4.1.2 20071124 (Red Hat 4.1.2-41)

*Disassembled the beam.smp, instruction pointer where the fault occurred
shown below.

 cat beam.smp.objdump | grep -C 10 4ee448
  4ee3fe:       48 3b 54 24 08          cmp    0x8(%rsp),%rdx
  4ee403:       48 89 74 24 68          mov    %rsi,0x68(%rsp)
  4ee408:       74 12                   je     4ee41c
<erts_garbage_collect+0x5ac>
  4ee40a:       48 8b 35 87 21 37 00    mov    3613063(%rip),%rsi        #
860598 <erts_allctrs+0x58>
  4ee411:       bf 09 00 00 00          mov    $0x9,%edi
  4ee416:       ff 15 74 21 37 00       callq  *3613044(%rip)        #
860590 <erts_allctrs+0x50>
  4ee41c:       4d 85 ff                test   %r15,%r15
  4ee41f:       0f 84 b1 08 00 00       je     4eecd6
<erts_garbage_collect+0xe66>
  4ee425:       48 8b 54 24 58          mov    0x58(%rsp),%rdx
  4ee42a:       48 39 54 24 68          cmp    %rdx,0x68(%rsp)
  4ee42f:       75 17                   jne    4ee448
<erts_garbage_collect+0x5d8>
  4ee431:       eb 52                   jmp    4ee485
<erts_garbage_collect+0x615>
  4ee433:       48 83 f8 02             cmp    $0x2,%rax
  4ee437:       0f 84 dc 05 00 00       je     4eea19
<erts_garbage_collect+0xba9>
  4ee43d:       48 83 c2 08             add    $0x8,%rdx
  4ee441:       48 39 54 24 68          cmp    %rdx,0x68(%rsp)
  4ee446:       74 3d                   je     4ee485
<erts_garbage_collect+0x615>
  4ee448:       48 8b 32                mov    (%rdx),%rsi
  4ee44b:       48 89 f0                mov    %rsi,%rax
  4ee44e:       83 e0 03                and    $0x3,%eax
  4ee451:       48 83 f8 01             cmp    $0x1,%rax
  4ee455:       0f 84 eb 04 00 00       je     4ee946
<erts_garbage_collect+0xad6>
  4ee45b:       73 d6                   jae    4ee433
<erts_garbage_collect+0x5c3>
  4ee45d:       48 89 f0                mov    %rsi,%rax
  4ee460:       83 e0 3c                and    $0x3c,%eax
  4ee463:       74 d8                   je     4ee43d
<erts_garbage_collect+0x5cd>
  4ee465:       48 83 f8 04             cmp    $0x4,%rax
  4ee469:       4c 8d 4a 08             lea    0x8(%rdx),%r9
  4ee46d:       0f 1f 00                nopl   (%rax)
  4ee470:       0f 84 5f 07 00 00       je     4eebd5
<erts_garbage_collect+0xd65>
  4ee476:       48 c1 ee 06             shr    $0x6,%rsi
  4ee47a:       49 8d 14 f1             lea    (%r9,%rsi,8),%rdx
  4ee47e:       48 39 54 24 68          cmp    %rdx,0x68(%rsp)
  4ee483:       75 c3                   jne    4ee448
<erts_garbage_collect+0x5d8>
  4ee485:       48 8b bd 20 02 00 00    mov    0x220(%rbp),%rdi
  4ee48c:       48 39 fb                cmp    %rdi,%rbx
  4ee48f:       0f 87 27 08 00 00       ja     4eecbc
<erts_garbage_collect+0xe4c>
  4ee495:       48 8b b5 10 02 00 00    mov    0x210(%rbp),%rsi
  4ee49c:       48 39 75 10             cmp    %rsi,0x10(%rbp)
  4ee4a0:       48 8b 44 24 58          mov    0x58(%rsp),%rax
  4ee4a5:       48 0f 44 44 24 68       cmove  0x68(%rsp),%rax
  4ee4ab:       48 83 bd 38 02 00 00    cmpq   $0x0,0x238(%rbp)
  4ee4b2:       00
  4ee4b3:       48 89 9d 20 02 00 00    mov    %rbx,0x220(%rbp)

Any help or knowledge would be appreciated. Thanks all

Regards,
Steven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150513/1b45109e/attachment.htm>