[erlang-bugs] beam core'ing

Patrik Nyblom <>
Wed Nov 21 12:09:28 CET 2012


Hi again :)

Another thing that would be helpful is if you could create a crash dump 
instead of a fprintf when the binary is wrongly moved, i.e. call 
erl_exit(ERTS_DUMP_EXIT, "erts_current_bin != (pb->bytes)"); instead of 
the fprintf? Then you could isolate the erlang code snippet that 
exercises the bug and I maybe could create a smaller testcase... A 
simple testcase when diving into the GC would be really helpful :)

Cheers,
/Patrik

On 11/21/2012 11:21 AM, Denis Titoruk wrote:
>
> 21.11.2012, в 13:44, Patrik Nyblom написал(а):
>
>> Hi!
>> On 11/20/2012 10:40 PM, Denis Titoruk wrote:
>>> Hi,
>>>
>>> We've got the same error on R15B01, R15B02
>>> I've finished my investigation of this issue today & here is result:
>>>
>>> Let's assume we have the code:
>>> encode_formats(Columns) ->
>>>    encode_formats(Columns, 0, <<>>).
>>>
>>> encode_formats([], Count, Acc) ->
>>>    <<Count:?int16, Acc/binary>>;
>>>
>>> encode_formats([#column{format = Format} | T], Count, Acc) ->
>>>    encode_formats(T, Count + 1, <<Acc/binary, Format:?int16>>).
>>>
>>> So, <<Acc/binary, Format:?int16>> translates to
>>>
>>>  {bs_append,{f,0},{integer,16},0,7,8,{x,2},{field_flags,[]},{x,1}}.
>>>    {bs_put_integer,{f,0},{integer,16},1,{field_flags,[signed,big]},{x,6}}.
>>>
>>> There is GC execution in bs_append and it can reallocate binary but 
>>> there isn't reassigning erts_current_bin which used in bs_put_integer.
>>>
>>> Fix:
>>>
>>> erl_bits.c:
>>> Eterm
>>> erts_bs_append(Process* c_p, Eterm* reg, Uint live, Eterm 
>>> build_size_term,
>>>        Uint extra_words, Uint unit)
>>>>>>    if (c_p->stop - c_p->htop < heap_need) {
>>>        (void) erts_garbage_collect(c_p, heap_need, reg, live+1);
>>>    }
>>>    sb = (ErlSubBin *) c_p->htop;
>>>    c_p->htop += ERL_SUB_BIN_SIZE;
>>>    sb->thing_word = HEADER_SUB_BIN;
>>>    sb->size = BYTE_OFFSET(used_size_in_bits);
>>>    sb->bitsize = BIT_OFFSET(used_size_in_bits);
>>>    sb->offs = 0;
>>>    sb->bitoffs = 0;
>>>    sb->is_writable = 1;
>>>    sb->orig = reg[live];
>>>
>>> ///////////////////////////////////////////////////////////////////
>>> // add this lines
>>> ///////////////////////////////////////////////////////////////////
>>>    pb = (ProcBin *) boxed_val(sb->orig);
>>>    erts_current_bin = pb->bytes;
>>>    erts_writable_bin = 1;
>>> ///////////////////////////////////////////////////////////////////
>>>
>>>    return make_binary(sb);
>>>>>>
>> Can you reproduce the bug and verify that this fix really works? The 
>> thing is that binaries should *only* be reallocated in the gc if 
>> there are no active writers, which there obviously is here (    
>> pb->flags |= PB_ACTIVE_WRITER  a few lines earlier), so the bug would 
>> be in the detection of active writers in the gc if this code change 
>> actually removes the crash.
>
> Yes, it works in my case. I haven't simple test case for reproducing 
> this bug (actually I run few processes to send requests to pgsql)
>
>     pb = (ProcBin *) boxed_val(sb->orig);
>     if (erts_current_bin != (pb->bytes)) {
>         fprintf(stderr, "erts_current_bin != (pb->bytes)\n");
>         fflush(stderr);
>     }
>     erts_current_bin = pb->bytes;
>     erts_writable_bin = 1;
>
>
> ()1> f(F), F = fun() -> postgresql:equery('echo-customers', 
> write, <<"some query here">>, []) end.
> #Fun<erl_eval.20.82930912>
> ()2> perftest:comprehensive(1000, F).
> Sequential 100 cycles in ~1 seconds (100 cycles/s)
> Sequential 200 cycles in ~2 seconds (106 cycles/s)
> Sequential 1000 cycles in ~12 seconds (85 cycles/s)
> Parallel 2 1000 cycles in ~8 seconds (132 cycles/s)
> Parallel 4 1000 cycles in ~8 seconds (121 cycles/s)
> Parallel 10 1000 cycles in ~8 seconds (119 cycles/s)
> Parallel 100 1000 cycles in ~13 seconds (74 cycles/s)
> [85,132,121,119,74]
> ()3> perftest:comprehensive(1000, F).
> Sequential 100 cycles in ~1 seconds (83 cycles/s)
> Sequential 200 cycles in ~2 seconds (83 cycles/s)
> Sequential 1000 cycles in ~14 seconds (71 cycles/s)
> Parallel 2 1000 cycles in ~11 seconds (95 cycles/s)
> Parallel 4 1000 cycles in ~10 seconds (105 cycles/s)
> Parallel 10 1000 cycles in ~11 seconds (91 cycles/s)
> Parallel 100 1000 cycles in ~13 seconds (76 cycles/s)
> "G_i[L"
> ()4> perftest:comprehensive(1000, F).
> Sequential 100 cycles in ~1 seconds (88 cycles/s)
> Sequential 200 cycles in ~2 seconds (85 cycles/s)
> Sequential 1000 cycles in ~13 seconds (74 cycles/s)
> Parallel 2 1000 cycles in ~9 seconds (109 cycles/s)
> Parallel 4 1000 cycles in ~10 seconds (101 cycles/s)
> Parallel 10 1000 cycles in ~11 seconds (95 cycles/s)
> erts_current_bin != (pb->bytes)
> Parallel 100 1000 cycles in ~13 seconds (77 cycles/s)
> "Jme_M"
>
>>
>>>
>>> --
>>> Cheers,
>>> Denis
>> Cheers,
>> /Patrik
>>>
>>> 20.11.2012, в 19:37, Musumeci, Antonio S написал(а):
>>>
>>>>
>>>> I've got lots of cores... but they are all from optimized builds.
>>>>
>>>> Has this been seen in other versions? We are keen to solve this 
>>>> because it's causing us pain in production. We hit another, older, 
>>>> memory bug (the 32bit values used in 64bit build)... and now this.
>>>>
>>>> I'm going to be building and trying R15B01 to see if we hit it as 
>>>> well. I'll send any additional information I can.Any suggestions on 
>>>> debugging beam would be appreciated. Compile options, etc.
>>>>
>>>> Thanks.
>>>>
>>>> -antonio
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* 
>>>> <mailto:>[mailto:]*On 
>>>> Behalf Of*Patrik Nyblom
>>>> *Sent:*Monday, November 19, 2012 8:55 AM
>>>> *To:* <mailto:>
>>>> *Subject:*Re: [erlang-bugs] beam core'ing
>>>>
>>>> On 11/19/2012 02:01 PM, Musumeci, Antonio S wrote:
>>>>>
>>>>> I'm just starting to debug this but figured I'd send it along in 
>>>>> case anyone has seen this before.
>>>>>
>>>>> 64bit RHEL 5.0.1
>>>>>
>>>>> built from source beam.smp R15B02
>>>>>
>>>>> Happens consistently when trying to start our app and then just 
>>>>> stops after a time. Across a few boxes. Oddly we have an identical 
>>>>> cluster (hw and sw) and it never happens.
>>>>>
>>>> Yes! I've seen it before and have tried for several months to get 
>>>> areproducable example and acore i can analyze here. I've had one 
>>>> core that wassomewhat readable but had no luck in locating the beam 
>>>> code that triggered this. If you could try narrowing it down, I 
>>>> would be really grateful!
>>>>
>>>> Please email me any findings, theories, cores dumps- anything! I 
>>>> really want to find this! The most interesting would be to find the 
>>>> snippet of erlang code that makes this happen (intermittently 
>>>> probably).
>>>>
>>>> The problem isthatwhen the allocators crash, the error is usually 
>>>> somewhere else.Access of freed memory, double free or something 
>>>> else doing horrid things to memory. Obviously none of our 
>>>> testsuites exercise this bug asneither our debug builds, nor our 
>>>> valgrind runs find it. It happens on both SMP and non SMP and is 
>>>> always in the context of the erts_bs_append, so I'm pretty sure 
>>>> this has a connection to the other users seeing the crash in the 
>>>> allocators...
>>>>
>>>> Cheers,
>>>> Patrik
>>>>>
>>>>> #0 bf_unlink_free_block (flags=<optimized out>, block=0x6f00, 
>>>>> allctr=<optimized out>) at beam/erl_bestfit_alloc.c:789
>>>>> #1 bf_get_free_block (allctr=0x6824600, size=304, cand_blk=0x0, 
>>>>> cand_size=<optimized out>, flags=0) at beam/erl_bestfit_alloc.c:869
>>>>> #2 0x000000000045343c in mbc_alloc_block (alcu_flgsp=<optimized 
>>>>> out>, blk_szp=<optimized out>, size=<optimized out>, 
>>>>> allctr=<optimized out>) at beam/erl_alloc_util.c:1198
>>>>> #3 mbc_alloc (allctr=0x6824600, size=295) at 
>>>>> beam/erl_alloc_util.c:1345
>>>>> #4 0x000000000045398d in do_erts_alcu_alloc (type=164, 
>>>>> extra=0x6824600, size=295) at beam/erl_alloc_util.c:3442
>>>>> #5 0x0000000000453a0f in erts_alcu_alloc_thr_pref (type=164, 
>>>>> extra=<optimized out>, size=287) at beam/erl_alloc_util.c:3520
>>>>> #6 0x0000000000511463 in erts_alloc (size=287, type=<optimized 
>>>>> out>) at beam/erl_alloc.h:208
>>>>> #7 erts_bin_nrml_alloc (size=<optimized out>) at beam/erl_binary.h:260
>>>>> #8 erts_bs_append (c_p=0x69fba60, reg=<optimized out>, 
>>>>> live=<optimized out>, build_size_term=<optimized out>, 
>>>>> extra_words=0, unit=8)at beam/erl_bits.c:1327
>>>>> #9 0x000000000053ffd8 in process_main () at beam/beam_emu.c:3858
>>>>> #10 0x00000000004ae853 in sched_thread_func (vesdp=<optimized 
>>>>> out>) at beam/erl_process.c:5184
>>>>> #11 0x00000000005c17e9 in thr_wrapper (vtwd=<optimized out>) at 
>>>>> pthread/ethread.c:106
>>>>> #12 0x00002b430f39e73d in start_thread () from /lib64/libpthread.so.0
>>>>> #13 0x00002b430f890f6d in clone () from /lib64/libc.so.6
>>>>> #14 0x0000000000000000 in ?? ()
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> NOTICE: Morgan Stanley is not acting as a municipal advisor and 
>>>>> the opinions or views contained herein are not intended to be, and 
>>>>> do not constitute, advice within the meaning of Section 975 of the 
>>>>> Dodd-Frank Wall Street Reform and Consumer Protection Act. If you 
>>>>> have received this communication in error, please destroy all 
>>>>> electronic and paper copies and notify the sender immediately. 
>>>>> Mistransmission is not intended to waive confidentiality or 
>>>>> privilege. Morgan Stanley reserves the right, to the extent 
>>>>> permitted under applicable law, to monitor electronic 
>>>>> communications. This message is subject to terms available at the 
>>>>> following link:http://www.morganstanley.com/disclaimersIf you 
>>>>> cannot access these links, please notify us by reply message and 
>>>>> we will send the contents to you. By messaging with Morgan Stanley 
>>>>> you consent to the foregoing.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> erlang-bugs mailing list
>>>>> 
>>>>> http://erlang.org/mailman/listinfo/erlang-bugs
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the 
>>>> opinions or views contained herein are not intended to be, and do 
>>>> not constitute, advice within the meaning of Section 975 of the 
>>>> Dodd-Frank Wall Street Reform and Consumer Protection Act. If you 
>>>> have received this communication in error, please destroy all 
>>>> electronic and paper copies and notify the sender immediately. 
>>>> Mistransmission is not intended to waive confidentiality or 
>>>> privilege. Morgan Stanley reserves the right, to the extent 
>>>> permitted under applicable law, to monitor electronic 
>>>> communications. This message is subject to terms available at the 
>>>> following link:http://www.morganstanley.com/disclaimersIf you 
>>>> cannot access these links, please notify us by reply message and we 
>>>> will send the contents to you. By messaging with Morgan Stanley you 
>>>> consent to the foregoing.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> erlang-bugs mailing list
>>>>  <mailto:>
>>>> http://erlang.org/mailman/listinfo/erlang-bugs
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20121121/85ae7148/attachment-0001.html>


More information about the erlang-bugs mailing list