[erlang-patches] erlang node crashes in erts_gc_after_bif_call

Patrik Nyblom <>
Fri Oct 19 17:08:37 CEST 2012


Hi!

On 10/19/2012 10:42 AM, adam chan wrote:
> Hi!
>
> I don't think the crypto NIF library was reloaded, though the mysql 
> client(Author: Magnus Ahltorp <>) in my project 
> does use crypto library.
>
> When I upgrade a big data file(600K) which only includes thousands of 
> lines like:
>      get(xxx) -> #record{a = xx, b = xx};
> , the crash happens.  I guess this data file has no relationship with 
> crypto NIF library?
>
Oups, that sure does not look like a crypto thing!
> Especially, while I execute
>      c:l(data_file_name)
> repeatedly and quickly in the screen shell, the crash shows up frequently.
Could you send me the file and an example program + your distro/os 
environment, so I can reproduce it?
Send it in a "private" mail to , you would maybe not want 
to spread details of your system publicly...
>
> Yesterday,  I found that the stack memory size of my application has 
> not been set, which means, it was running on the linux default stack 
> size (10M).  After I set the stack size to 500M using 'ulimit -s ' 
> command, and split the big  data file into small sub files, the 
> situation becomes better. Maybe the small stack size is the criminal, 
> but I am not sure.   : (
>
Nah, that should not be a problem as long as you do not use external 
libraries which go crazy on the C stack. We usually don't (except for 
re, which relies on the PCRE library, that has a rather aggressive 
approach to the C stack when compiling regexps).
> After all, is there any way to detect whether the crypto NIF library 
> is reloaded or not?
> I've found a discussion about "fix native code crash when calling 
> unloaded module with on_load function":
>     
>  http://erlang.2086793.n4.nabble.com/fix-native-code-crash-when-calling-unloaded-module-with-on-load-function-td2273502.html
> And I did have a suspicion on crypto module before, since the crypto 
> module has an on_load attribute.
In a few days, there will be a fix for this in the master branch, that 
should remove the problem. But if you have an example you can run from 
the shell and that I could reproduce it with, things would be much 
faster. You could of course add a printout to the crypto module to track 
the calls of the on_load handler. If it's called more than twice, there 
will also be unloading (because of code purging). You could also trace 
calls to erlang:purge_module/1.
>
> Cheers,
> [Adam Chan]
Cheers,
/Patrik
>
> ------------------ Original ------------------
> *From: * "Patrik Nyblom"<>;
> *Date: * Thu, Oct 18, 2012 08:08 PM
> *To: * "erlang-patches"<>;
> *Subject: * Re: [erlang-patches] erlang node crashes in 
> erts_gc_after_bif_call
>
> Hi!
>
> Is the crypto NIF library reloaded during upgrade? That causes havoc
> unfortunately, due to the behaviour of the OpenSSL crypto memory
> allocation callbacks. We're working on that one.
>
> Have you reloaded the crypto NIF library, directly or indirectly, when
> this happens?
>
> Cheers,
> /Patrik
>
> On 10/17/2012 03:47 AM, adam chan wrote:
> > hello list,
> >
> >      I met two random crash in this month, each crash happened more 
> than two
> > times. The causation was "Program terminated with signal 11, 
> Segmentation
> > fault" and they most likely happened while I hot update some module code
> > using code:soft_purge/1 and code:load_file/1.
> >      Though they take place in different code, the information from 
> core file
> > points out that function erts_gc_after_bif_call/4 was called while crash
> > happened. So I guess it is related to gc operation.
> >      I am using otp_src_R15B02, smp mode.
> >      Here are the information from core file (I am not familiar with 
> gdb ,
> > hope the information is useful)
> >
> >      [First One]
> > Reading symbols from /lib64/libutil.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libutil.so.1
> > Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libdl.so.2
> > Reading symbols from /lib64/libm.so.6...(no debugging symbols 
> found)...done.
> > Loaded symbols for /lib64/libm.so.6
> > Reading symbols from /usr/lib64/libncurses.so.5...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/lib64/libncurses.so.5
> > Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libpthread.so.0
> > Reading symbols from /lib64/librt.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/librt.so.1
> > Reading symbols from /lib64/libc.so.6...(no debugging symbols 
> found)...done.
> > Loaded symbols for /lib64/libc.so.6
> > Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/ld-linux-x86-64.so.2
> > Reading symbols from
> > /usr/local/lib/erlang/lib/crypto-2.2/priv/lib/crypto.so...done.
> > Loaded symbols for 
> /usr/local/lib/erlang/lib/crypto-2.2/priv/lib/crypto.so
> > Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libcrypto.so.6
> > Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/lib64/libz.so.1
> > Core was generated by `/usr/local/lib/erlang/erts-5.9.2/bin/beam.smp -P
> > 1024000 -K true -- -root /usr/'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x0000000000541b9a in check_process_code (A__p=0x1a8c5b40,
> > BIF__ARGS=<value optimized out>) at beam/beam_bif_load.c:487
> > 487                 if (INSIDE((BeamInstr *) funp->fe->address)) {
> > (gdb) bt
> > #0  0x0000000000541b9a in check_process_code (A__p=0x1a8c5b40,
> > BIF__ARGS=<value optimized out>) at beam/beam_bif_load.c:487
> > #1  check_process_code_2 (A__p=0x1a8c5b40, BIF__ARGS=<value 
> optimized out>)
> > at beam/beam_bif_load.c:205
> > #2  0x0000000000530782 in process_main () at beam/beam_emu.c:3392
> > #3  0x00000000004a0b4f in sched_thread_func (vesdp=<value optimized 
> out>) at
> > beam/erl_process.c:5184
> > #4  0x00000000005a4f14 in thr_wrapper (vtwd=<value optimized out>) at
> > pthread/ethread.c:110
> > #5  0x000000393fc0673d in start_thread () from /lib64/libpthread.so.0
> > #6  0x000000393f4d3f6d in clone () from /lib64/libc.so.6
> > (gdb) p funp
> > $1 =<value optimized out>
> > (gdb) p funp->fe
> > Cannot access memory at address 0x8
> >
> >      [Second One]
> > Reading symbols from /lib64/libutil.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libutil.so.1
> > Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libdl.so.2
> > Reading symbols from /lib64/libm.so.6...(no debugging symbols 
> found)...done.
> > Loaded symbols for /lib64/libm.so.6
> > Reading symbols from /usr/lib64/libncurses.so.5...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/lib64/libncurses.so.5
> > Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libpthread.so.0
> > Reading symbols from /lib64/librt.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/librt.so.1
> > Reading symbols from /lib64/libc.so.6...(no debugging symbols 
> found)...done.
> > Loaded symbols for /lib64/libc.so.6
> > Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/ld-linux-x86-64.so.2
> > Reading symbols from
> > /usr/local/lib/erlang/lib/crypto-2.2/priv/lib/crypto.so...done.
> > Loaded symbols for 
> /usr/local/lib/erlang/lib/crypto-2.2/priv/lib/crypto.so
> > Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libcrypto.so.6
> > Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/lib64/libz.so.1
> > Core was generated by `/usr/local/lib/erlang/erts-5.9.2/bin/beam.smp -P
> > 1024000 -K true -- -root /usr/'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x00000000004f8bac in sweep_off_heap (p=0x2aaabc644b90, 
> fullsweep=0) at
> > beam/erl_gc.c:2302
> > 2302                ptr = ptr->next;
> > (gdb) bt
> > #0  0x00000000004f8bac in sweep_off_heap (p=0x2aaabc644b90, 
> fullsweep=0) at
> > beam/erl_gc.c:2302
> > #1  0x00000000004fabb8 in do_minor (p=0x2aaabc644b90, need=0,
> > objv=0x44719e00, nobj=1, recl=0x44719db8) at beam/erl_gc.c:1133
> > #2  minor_collection (p=0x2aaabc644b90, need=0, objv=0x44719e00, nobj=1,
> > recl=0x44719db8) at beam/erl_gc.c:827
> > #3  0x00000000004fc40d in erts_garbage_collect (p=0x2aaabc644b90, 
> need=0,
> > objv=0x44719e00, nobj=1) at beam/erl_gc.c:405
> > #4  0x00000000004fcdcf in erts_gc_after_bif_call (p=0x2aaabc644b90,
> > result=46912734893994, regs=<value optimized out>, arity=<value 
> optimized
> > out>) at beam/erl_gc.c:335
> > #5  0x00000000005309c1 in process_main () at beam/beam_emu.c:2600
> > #6  0x00000000004a0b4f in sched_thread_func (vesdp=<value optimized 
> out>) at
> > beam/erl_process.c:5184
> > #7  0x00000000005a4f14 in thr_wrapper (vtwd=<value optimized out>) at
> > pthread/ethread.c:110
> > #8  0x000000393fc0673d in start_thread () from /lib64/libpthread.so.0
> > #9  0x000000393f4d3f6d in clone () from /lib64/libc.so.6
> > (gdb) p ptr
> > $1 = (struct erl_off_heap_header *) 0x2aab02d63980
> > (gdb) x/x 0x2aab02d63980
> > 0x2aab02d63980: 0x000000f0
> > (gdb) p ptr->next
> > $2 = (struct erl_off_heap_header *) 0x2aab02d5ee80
> > (gdb) x/x 0x2aab02d5ee80
> > 0x2aab02d5ee80: 0x00000160
> >
> >      Any ideas? Thanks a lot.
> >
> >
> >
> > --
> > View this message in context: 
> http://erlang.2086793.n4.nabble.com/erlang-node-crashes-in-erts-gc-after-bif-call-tp4655148.html
> > Sent from the Erlang Patches mailing list archive at Nabble.com.
> > _______________________________________________
> > erlang-patches mailing list
> > 
> > http://erlang.org/mailman/listinfo/erlang-patches
>
> _______________________________________________
> erlang-patches mailing list
> 
> http://erlang.org/mailman/listinfo/erlang-patches
>
>
> _______________________________________________
> erlang-patches mailing list
> 
> http://erlang.org/mailman/listinfo/erlang-patches

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20121019/49644a4a/attachment-0001.html>


More information about the erlang-patches mailing list