It's kind of cheating, as it's not documented.(AFAIK)<div>x is euqal to </div><div>calling dbg:fun2ms(fun(_)-> exception_trace() end). </div><div>which generates MatchSpec : </div><div>[{'_',[],[{exception_trace}]}]</div>
<div><br></div><div>from documentation of match specs</div><div><a href="http://www.erlang.org/doc/apps/erts/match_spec.html">http://www.erlang.org/doc/apps/erts/match_spec.html</a></div><div><strong style="font-family: Verdana, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); ">exception_trace:</strong><span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); "> Same as </span><strong style="font-family: Verdana, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); ">return_trace</strong><span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); ">, plus; if the traced function exits due to an exception, an</span><span class="code" style="font-family: courier; background-color: rgb(255, 255, 255); ">exception_from</span><span class="Apple-style-span" style="font-family: Verdana, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); "> trace message is generated, whether the exception is caught or not.</span></div>
<div><br></div><div>example: </div><div><div>37> dbg:tracer(). </div><div>{ok,<0.85.0>}</div><div>38> dbg:tpl(calendar, x). </div><div>{ok,[{matched,nonode@nohost,39},{saved,x}]}</div>
<div>39> dbg:p(all,c). </div><div>{ok,[{matched,nonode@nohost,26}]}</div><div>40> calendar:local_time_to_universal_time(33).</div><div>(<0.83.0>) call calendar:local_time_to_universal_time(33)</div>
<div>(<0.83.0>) exception_from {calendar,local_time_to_universal_time,1} {error,badarg}</div><div>** exception error: bad argument</div><div> in function erlang:localtime_to_universaltime/2</div><div> called as erlang:localtime_to_universaltime(33,undefined)</div>
<div> in call from erlang:localtime_to_universaltime/1</div></div><div><br></div><div><br></div><div>I think better use the documented form :)</div><div>dbg:tpl(usr_drv, dbg:fun2ms(fun(_)-> exception_trace() end)).</div>
<div><br></div><div><br><div class="gmail_quote">On Mon, Nov 28, 2011 at 4:38 PM, Kirill Zaborsky <span dir="ltr"><<a href="mailto:qrilka@gmail.com">qrilka@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Ahmed, minor question: what does 'x' mean in dbg:tpl(user_drv, x) ? From the documentation I do not see what this match specification mya mean.<div><br></div><div>Kind regards,</div><div><span class="HOEnZb"><font color="#888888">Kirill Zaborsky</font></span><div>
<div class="h5"><br><br><div class="gmail_quote">
2011/11/28 Ahmed Omar <span dir="ltr"><<a href="mailto:spawn.think@gmail.com" target="_blank">spawn.think@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Krill, <div>How about using some dbg? </div><div>Note : you need to be careful not to generate huge amount of debugging on your node that could actually kill your node. I don't know much about your system and how heavy the load on nodes is.</div>
<div><br></div><div>You can do something like : </div><div>dbg:tracer(port, dbg:trace_port(file, "/tmp/usr_drv.dbg")).</div><div>dbg:p(user_drv, [m, c]).</div><div>dbg:tpl(user_drv, x).</div><div><br></div><div>
This way you will have a file that contains tracing of all messages and local function calls done in process user_drv</div><div>When the node stuck we can see more details about what happened (note : watch for the file size too)<br>
<br><div class="gmail_quote"><div><div>On Mon, Nov 28, 2011 at 1:37 PM, Kirill Zaborsky <span dir="ltr"><<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>></span> wrote:<br></div></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
Thanks, dennis,<div>I have created a crash dump on a test machine (using halfword emulator) and received user_drv in waiting state with</div><div>Program counter - 0x0000000002845f00 (user_drv:server_loop/5 + 48)</div><div>
So it's on the same instruction (but not running)</div><div><br></div><div>Disassembly shows:</div><div>-----------------</div><div><div>0000000002845E80: i_func_info_IaaI 0 user_drv server_loop 5 </div><div>0000000002845EA8: allocate_init_tIy 7 5 y(0) </div>
<div>0000000002845EC0: init_y y(1) </div><div>0000000002845ED0: move2_xyxy x(4) y(2) x(3) y(3) </div><div>0000000002845EE0: move2_xyxy x(2) y(4) x(1) y(5) </div><div>0000000002845EF0: move_ry x(0) y(6) </div><div>0000000002845F00: i_loop_rec_fr f(0000000002846C80) x(0) </div>
<div>0000000002845F10: i_select_tuple_arity2_rfAfAf x(0) f(0000000002846C40) 2 f(0000000002845F40) 3 f(0000000002846418) </div><div>0000000002845F40: i_get_tuple_element_rPx x(0) 0 x(1) </div><div>.....</div><div><div>0000000002846C80: wait_f f(0000000002845F00) </div>
<div>0000000002846C90: badmatch_r x(0) </div></div><div>-----------------</div><div>So it's just a waiting loop. I don'see how the process could be running when the only ouput for some time was "ALIVE" messages every 15 minutes from run_erl.</div>
<div>Loooks like the only way to see what was going on is to get complete crash dump, but it was truncated by heart :-\</div><div><br></div><div>P.S. It's quite strange that crash dump shows +48</div><br><div class="gmail_quote">
2011/11/28 <span dir="ltr"><<a href="mailto:dennis.novikov@gmail.com" target="_blank">dennis.novikov@gmail.com</a>></span><div><div><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+48 does not point to an instruction start on a couple of 32-bit systems I have access to, so I can not assist you further.<br>
<br>
To get instructions dump named "user_drv.dis" in the beam process working directory you can do<br>
<br>
erts_debug:df(user_drv).<br>
<br>
Happy bug-hunting.<div><div><br>
<br>
<br>
On Mon, 28 Nov 2011 12:01:17 +0200, Kirill Zaborsky <<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm using halfword emulator on 64bit Ubuntu Server<br>
And the process state is not "waiting" but "running". Previous crash dumps<br>
show the same program counter value (and user_drv in running state)<br>
<br>
Kind regards,<br>
Kirill Zaborsky<br>
<br>
<br>
2011/11/28 Dennis Novikov <<a href="mailto:dennis.novikov@gmail.com" target="_blank">dennis.novikov@gmail.com</a>><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Mon, 28 Nov 2011 08:44:42 +0200, Kirill Zaborsky <<a href="mailto:qrilka@gmail.com" target="_blank">qrilka@gmail.com</a>><br>
wrote:<br>
<br>
Trying to fins any workaround to this "stuck node" scenario I've upgraded<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
to R14B04 and turned on "heart".<br>
But recently the node once again stopped responding. And heart did not<br>
assume it to be stuck although I could not contact it.<br>
I've tried to to get a crashdump with 'kill -USR1' but it appeared that<br>
once again crash dump was truncated. Does heart kills "dead" erlang node?<br>
And the only thing that could be seen from the crash dump that the only<br>
running process was user_drv (just like in previous times) with program<br>
counter equal to "user_drv:server_loop/5 + 48". Is it possible to find out<br>
what exactly does it stands for?<br>
<br>
</blockquote>
<br>
Waiting on receive in that function. And you are observing this on a<br>
32-bit VM.<br>
<br>
--<br>
WBR,<br>
DN<br>
<br>
</blockquote></blockquote>
<br>
<br></div></div><span><font color="#888888">
-- <br>
WBR,<br>
DN<br>
</font></span></blockquote></div></div></div><br></div>
<br></div></div><div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></div></blockquote></div><div><br><br clear="all"><div><br></div>-- <br>Best Regards,<br>- Ahmed Omar<div><a href="http://nl.linkedin.com/in/adiaa" target="_blank">http://nl.linkedin.com/in/adiaa</a></div>
<div>Follow me on twitter</div>
<div><a href="http://twitter.com/#!/spawn_think" target="_blank">@spawn_think</a></div><br>
</div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Best Regards,<br>- Ahmed Omar<div><a href="http://nl.linkedin.com/in/adiaa" target="_blank">http://nl.linkedin.com/in/adiaa</a></div><div>Follow me on twitter</div>
<div><a href="http://twitter.com/#!/spawn_think" target="_blank">@spawn_think</a></div><br>
</div>