<div dir="ltr">Hi Lukas, <br><div>Is it possible to use the etp-commands on a core file or gdb has to be attached to a live node?</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Best Regards,<br>- Ahmed Omar<div><span style="color:rgb(51,51,51);font-family:proxima-nova-1,proxima-nova-2,Tahoma,Helvetica,Verdana,sans-serif;font-size:14px;line-height:18px"><a href="http://about.me/spawn.think/" target="_blank">http://about.me/spawn.think/</a></span></div></div></div></div>
<br><div class="gmail_quote">2015-09-08 14:54 GMT+02:00 Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com" target="_blank">spawn.think@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Lukas, <div><br></div><div>Thanks a lot for the pointers, I'll see if I can find anything.</div></div><div class="gmail_extra"><span class=""><br clear="all"><div><div><div dir="ltr">Best Regards,<br>- Ahmed Omar<div><span style="color:rgb(51,51,51);font-family:proxima-nova-1,proxima-nova-2,Tahoma,Helvetica,Verdana,sans-serif;font-size:14px;line-height:18px"><a href="http://about.me/spawn.think/" target="_blank">http://about.me/spawn.think/</a></span></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">2015-09-08 12:11 GMT+02:00 Lukas Larsson <span dir="ltr"><<a href="mailto:garazdawi@gmail.com" target="_blank">garazdawi@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>Unfortunately that did not help, it just made some more arguments available. I was hoping that it would give a full stack. If you do "p allctr->name_prefix" you can get to know which allocator it is that is miss behaving. I'm guessing that it will be "driver_", which is any allocations done in nifs+linked-in drivers. Without a full stacktrace it will be hard to figure out what is wrong.</div><div><br></div><div>One thing that you could do it to use the etp gdb macros distributed with the Erlang/OTP source code. If you in gdb do "source $ERL_TOP/erts/etc/unix/etp-commands" (replacing $ERL_TOP with the path to the source of R16B03-1 Erlang/OTP from github. It is only this file that is needed, so if you need to copy this onto a server somewhere you only need this file) you will get access to a lot of helpful gdb macros.</div><div><br></div><div>If you then do "etp-ports" you will get printed to the shell all ports that are alive at the moment of the crash. Look for any ports with a state that looks different. That will most likely be the port that is just executing. e.g. for me this is a currently running port:</div><div><br></div><div><div> Pix: 2576</div><div> Port: #Port<0.322></div><div> Name: tty_sl -c -e</div><div> State: connected soft-eof</div><div> Scheduler flags: GARBAGE</div><div> Connected: <0.25.0></div><div> Pointer: (Port *) 0x7ffff54809d8</div></div><div><br></div><div>to get the name of the currently running driver do "p ((Port *)0x7ffff54809d8)->drv_ptr->name".</div><div><br></div><div>If no port is executing, it might be a nif, then you can do "etp-processes" to get a list of all processes in the system. Again look for any state that looks different. e.g.</div><div><br></div><div><div> Pix: 200</div><div> Pid: <0.25.0></div><div> State: trapping-exit | running | active | prq-prio-normal | usr-prio-normal | act-prio-normal</div><div> Registered name: user_drv</div><div> I: #Cp<user_drv:io_command/1+0x520></div><div> Heap size: 610</div><div> Old-heap size: 987</div><div> Mbuf size: 0</div><div> Msgq len: 0 (inner=0, outer=0)</div><div> Parent: <0.24.0></div><div> Pointer: (Process *) 0x7ffff51c4df0</div></div><div><br></div><div>This is a currently running process (State: running). You can get a stackdump of the process by doing: etp-stackdump ((Process*)0x7ffff51c4df0)</div><div><br></div><div>Note that etp-processes and etp-ports will take quite some time to run until they finish. They need to iterate over all possibly processes/ports, and gdb is not the fastest scripting language in the world.</div><div><br></div><div>For some help with the etp gdb commands you can issue "etp-help".</div><div><br></div><div>Happy hunting!</div><span><font color="#888888"><div>Lukas</div></font></span><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 8, 2015 at 11:45 AM, Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com" target="_blank">spawn.think@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi Lukas, <div>Thanks for your reply. I tried with the latest version of gdb (7.10) :</div><div><br></div><div>###</div><div><span><div>(gdb) bt full</div><div>#0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) at beam/erl_goodfit_alloc.c:439</div><div> gfallctr = 0x15e32c0</div><div> blk = 0x128</div><div> sz = 0</div></span><div> i = <optimized out></div><span><div>#1 0x00000000015e32c0 in ?? ()</div><div>No symbol table info available.</div></span><div>#2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, size=<optimized out>, busy_pcrr_pp=0x8, alcu_flgs=0) at beam/erl_alloc_util.c:2370</div><div> crr = 0x128</div><div> new_p = <optimized out></div><span><div> old_blk_sz = 287</div><div> blk = 0x117</div></span><div> new_blk = <optimized out></div><div> cand_blk = <optimized out></div><div> cand_blk_sz = <optimized out></div><span><div> blk_sz = 3748409</div><div> nxt_blk = 0x236</div><div> nxt_blk_sz = 22950592</div><div> is_last_blk = 296</div><div> get_blk_sz = 140602277246336</div><div>#3 0x0000000000000000 in ?? ()</div><div>No symbol table info available.</div></span></div><div>###</div></div><div class="gmail_extra"><br clear="all"><span><div><div><div dir="ltr">Best Regards,<br>- Ahmed Omar<div><span style="color:rgb(51,51,51);font-family:proxima-nova-1,proxima-nova-2,Tahoma,Helvetica,Verdana,sans-serif;font-size:14px;line-height:18px"><a href="http://about.me/spawn.think/" target="_blank">http://about.me/spawn.think/</a></span></div></div></div></div>
<br></span><div><div><div class="gmail_quote">2015-09-08 10:51 GMT+02:00 Lukas Larsson <span dir="ltr"><<a href="mailto:garazdawi@gmail.com" target="_blank">garazdawi@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello,<br><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Tue, Sep 8, 2015 at 10:33 AM, Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com" target="_blank">spawn.think@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi, <div>We have been experiencing a segfault on our servers running a custom version of Ejabberd. We managed to get a core file from the last crash</div><div>This is what we see running gdb on it:</div><div>######</div><div><div>Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.</div><div>Loaded symbols for /lib64/ld-linux-x86-64.so.2</div><div>Core was generated by `/var/lib/ejabberd/erts-5.10.4/bin/beam.smp -K true -A 128 -P 2500000 -Q 500000'.</div><div>Program terminated with signal 11, Segmentation fault.</div><div>#0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) at beam/erl_goodfit_alloc.c:439</div><div>439<span style="white-space:pre-wrap"> </span>beam/erl_goodfit_alloc.c: No such file or directory.</div><div><span style="white-space:pre-wrap"> </span>in beam/erl_goodfit_alloc.c</div></div><div>######</div><div><br></div><div>If we run bt full in gdb we get:</div><div>######</div><div><div><div>(gdb) bt full</div><div>#0 0x000000000044d299 in link_free_block (allctr=0x15e32c0, block=0x128) at beam/erl_goodfit_alloc.c:439</div><div> gfallctr = 0x15e32c0</div><div> blk = 0x128</div><div> sz = 0</div><div> i = <value optimized out></div><div>#1 0x00000000015e32c0 in ?? ()</div><div>No symbol table info available.</div><div>#2 0x0000000000442fa6 in mbc_realloc (allctr=0x7fe0848807a8, p=0x11f, size=Unhandled dwarf expression opcode 0xf3</div><div>) at beam/erl_alloc_util.c:2370</div><div> crr = 0x128</div><div> new_p = <value optimized out></div><div> old_blk_sz = 287</div><div> blk = 0x117</div><div> new_blk = <value optimized out></div><div> cand_blk = <value optimized out></div><div> cand_blk_sz = <value optimized out></div><div> blk_sz = 3748409</div><div> nxt_blk = 0x236</div><div> nxt_blk_sz = 22950592</div><div> is_last_blk = 296</div><div> get_blk_sz = 140602277246336</div><div>#3 0x0000000000000000 in ?? ()</div><div>No symbol table info available.</div></div><div>#######</div><div><br></div><div>Is there a way to get more information? maybe which driver made the realloc call?</div></div></div></blockquote><div><br></div></div></div><div>Something is wrong/missing from this stacktrace. The gdb that you are using does not seem to understand the dwarf2 extension (at least that's what I guess after googling "Unhandled dwarf expression opcode 0xf3"), and can only find two of the frames. Try to install a later version of gdb and then do a bt full.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><div dir="ltr"><div><div><br></div><div><div><div dir="ltr">Best Regards,<br>- Ahmed Omar<div><span style="color:rgb(51,51,51);font-family:proxima-nova-1,proxima-nova-2,Tahoma,Helvetica,Verdana,sans-serif;font-size:14px;line-height:18px"><a href="http://about.me/spawn.think/" target="_blank">http://about.me/spawn.think/</a></span></div></div></div></div>
</div></div>
<br></span>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>