<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB"> Li, Huiqing
<br>
<b>Sent:</b> 08 September 2014 21:20<br>
<b>To:</b> 'erlang questions'<br>
<b>Subject:</b> processes stuck in erlang:bif_return_trap/1<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Dear All, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I wonder if anyone could help me with this. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have been experiencing some problems with an Erlang node running Mnesia. For some reason, the node becomes unresponsive every few minutes, but the system does recover after a short period of time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We did some profiling using ETop, and it shows that when the system freezes, there are about 5000 processes accumulated on this node getting stuck in the erlang:bif_return_trap/1 function. These processes are spawned by a rpc_server running
on this node; each process computes a M:F(A) then sends the result back via gen_tcp or stores the result in an ets table.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don’t quite understand what erlang:bif_return_trap/1 does, and am confused why so many processes got stuck in this function. The Erlang node is running on a 12 physical core machine (hence 24 schedulers), and the version of Erlang is
R15B03. Here is part of the ETop output:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Load: cpu 317 Memory: total 66521736 binary 393593<o:p></o:p></p>
<p class="MsoNormal"> <span lang="FR">procs 5785 processes 1253072 code 12280<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="FR"> runq 0 atom 493 ets 64843631<o:p></o:p></span></p>
<p class="MsoNormal">Pid Name or Initial Func Time Reds Memory MsgQ Current Function<o:p></o:p></p>
<p class="MsoNormal"><span lang="IT">----------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="IT"><5291.115.0> mnesia_tm '-' 0******** 0 mnesia_tm:doit_loop/<o:p></o:p></span></p>
<p class="MsoNormal"><5291.6.0> error_logger '-' 012342496 0 gen_event:fetch_msg/<o:p></o:p></p>
<p class="MsoNormal"><5291.61.0> memsup '-' 138 4714912 0 gen_server:loop/6
<o:p></o:p></p>
<p class="MsoNormal"><5291.1464.0> proc_lib:init_p/5 '-' 103 2914384 0 gen_fsm:loop/7
<o:p></o:p></p>
<p class="MsoNormal"><5291.1466.0> proc_lib:init_p/5 '-' 105 2914384 0 gen_fsm:loop/7
<o:p></o:p></p>
<p class="MsoNormal"> .<o:p></o:p></p>
<p class="MsoNormal"> .<o:p></o:p></p>
<p class="MsoNormal"> .<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 8141 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 7900 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 7901 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 7954 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 7975 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 8068 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 7956 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal">***************rpc_socket:worker/6 '-' 8212 101144 0 erlang:bif_return_tr<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Many Thanks! <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Huiqing<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<P>NOTICE AND DISCLAIMER<BR>This e-mail (including any attachments) is intended
for the above-named person(s). If you are not the intended recipient,
notify the sender immediately, delete this email from your system and do not
disclose or use for any purpose. <BR> <BR>We may monitor all incoming
and outgoing emails in line with current legislation. We have taken steps to
ensure that this email and attachments are free from any virus, but it remains
your responsibility to ensure that viruses do not adversely affect you. </P>
<P>EE Limited<BR>Registered in England and Wales<BR>Company Registered Number:
02382161<BR>Registered Office Address: Trident Place, Mosquito Way, Hatfield,
Hertfordshire, AL10 9BW</P>
</body>
</html>