<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Thanks, I'll give that a go,</p>
<p><br>
</p>
<p>Looking at code_server.erl it does appear that there exists a possibility of a deadlock. In so much as function cpc_recv/4 does a receive with no timeout.</p>
<p><br>
</p>
<p>Thanks again</p>
<p><br>
</p>
<p>Matt</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Michael Truog <mjtruog@gmail.com><br>
<b>Sent:</b> Friday, December 9, 2016 1:03 PM<br>
<b>To:</b> Matthew Evans; Erlang/OTP discussions<br>
<b>Subject:</b> Re: [erlang-questions] R17 - Possible wedged scheduler</font>
<div> </div>
</div>
<div>
<div class="moz-cite-prefix">If this is scheduler collapse, it would mean you have a port driver or NIF that has internal latency greater than 1 millisecond. To handle scheduler collapse, you can use the erl command line option "-heart" combined with:<br>
heart:set_options([check_schedulers]).<br>
<br>
(see <a class="moz-txt-link-freetext" href="http://erlang.org/doc/man/heart.html#set_options-1" id="LPlnk218373" previewremoved="true">
http://erlang.org/doc/man/heart.html#set_options-1</a> )<br>
<br>
That will allow the system to restart when schedulers have collapsed.<br>
<br>
A test that is meant to cause scheduler collapse is at:<br>
<a class="moz-txt-link-freetext" href="https://github.com/basho/nifwait" id="LPlnk322925" previewremoved="true">https://github.com/basho/nifwait</a>
<div id="LPBorder_GT_14813081479850.6086475834243559" style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;">
<table id="LPContainer_14813081479840.32224113159410894" cellspacing="0" style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);">
<tbody>
<tr valign="top" style="border-spacing: 0px;">
<td id="ImageCell_14813081479840.13930987925918026" colspan="1" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;">
<div id="LPImageContainer_14813081479840.12966524762997267" style="background-color: rgb(255, 255, 255); height: 250px; position: relative; margin: auto; display: table; width: 250px;">
<a id="LPImageAnchor_14813081479840.9300885298841379" href="https://github.com/basho/nifwait" target="_blank" style="display: table-cell; text-align: center;"><img aria-label="Preview image with link selected. Double-tap to open the link." id="LPThumbnailImageID_14813081479840.552299767837289" width="250" height="250" style="display: inline-block; max-width: 250px; max-height: 250px; height: 250px; width: 250px; border-width: 0px; vertical-align: bottom;" src="https://avatars0.githubusercontent.com/u/176293?v=3&s=400"></a></div>
</td>
<td id="TextCell_14813081479850.20075065014414895" colspan="2" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;">
<div id="LPRemovePreviewContainer_14813081479850.846937679760593"></div>
<div id="LPTitle_14813081479850.7206280393934008" style="top: 0px; color: rgb(0, 120, 215); font-weight: normal; font-size: 21px; font-family: wf_segoe-ui_light, "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; line-height: 21px;">
<a id="LPUrlAnchor_14813081479850.14230599362913798" href="https://github.com/basho/nifwait" target="_blank" style="text-decoration: none;">GitHub - basho/nifwait: Utility to test effect of blocking ...</a></div>
<div id="LPMetadata_14813081479850.3841899577719021" style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: normal; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 14px; line-height: 14px;">
github.com</div>
<div id="LPDescription_14813081479850.9824006852799367" style="display: block; color: rgb(102, 102, 102); font-weight: normal; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 14px; line-height: 20px; max-height: 100px; overflow: hidden;">
README.md Utility to test effect of blocking NIF on Erlang scheduler. Test spawns several processes, all which will start in the run queue of the current scheduler.</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
<br>
<br>
So you could use that to prove to yourself that the behavior is the same with the system you are having problems with.<br>
<br>
<br>
On 12/09/2016 08:35 AM, Matthew Evans wrote:<br>
</div>
<blockquote type="cite">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Happened again, it appears that code_server is wedged:</p>
<p><br>
</p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style="">admin@plexxi:~$ doErlangFun "erlang:process_info(whereis(code_server))."</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style="">[{registered_name,code_server},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {current_function,{code_server,cpc_recv,4}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {initial_call,{erlang,apply,2}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {status,waiting},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {message_queue_len,23},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {messages,[{code_call,<6805.4097.0>,{ensure_loaded,switch_type_module}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.4146.0>,{ensure_loaded,switch_type_module}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.941.0>,{ensure_loaded,pc_port_autoneg}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.541.0>,{ensure_loaded,plexxiStatistics_types}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.520.0>,{ensure_loaded,switch_type_module}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.5123.0>,{ensure_loaded,secondary_erlang_node}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.5122.0>,{ensure_loaded,secondary_erlang_node}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.5162.0>,{ensure_loaded,icmp}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.5321.0>,</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {ensure_loaded,mac_entries_record_handler}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.5483.0>,{ensure_loaded,icmp}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.6647.0>,{ensure_loaded,icmp}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.7232.0>,{ensure_loaded,icmp}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.7274.0>,{ensure_loaded,icmp}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.7304.0>,{ensure_loaded,icmp}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.8889.0>,</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {ensure_loaded,mac_entries_record_handler}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.8951.0>,</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {ensure_loaded,mac_entries_record_handler}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.576.0>,</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {ensure_loaded,cross_connect_unicast_utils}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.19300.12>,{ensure_loaded,shell}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.20313.12>,{ensure_loaded,shell}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.21339.12>,{ensure_loaded,dbg}},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.31109.13>,get_mode},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {code_call,<6805.1255.14>,get_mode},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {system,{<6805.2521.14>,#Ref<6805.0.23.35356>},get_status}]},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {links,[<6805.11.0>]},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {dictionary,[{any_native_code_loaded,false}]},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {trap_exit,true},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {error_handler,error_handler},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {priority,normal},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {group_leader,<6805.9.0>},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {total_heap_size,86071},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {heap_size,10958},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {stack_size,25},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {reductions,13172282},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {garbage_collection,[{min_bin_vheap_size,46422},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {min_heap_size,233},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {fullsweep_after,65535},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {minor_gcs,71}]},</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style=""> {suspending,[]}]</span></p>
<p style="margin:0px; font-style:normal; font-weight:normal; font-size:11px; line-height:normal; font-family:Menlo">
<span style="">admin@plexxi:~$ </span></p>
<br>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions-bounces@erlang.org">
erlang-questions-bounces@erlang.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:erlang-questions-bounces@erlang.org">
<erlang-questions-bounces@erlang.org></a> on behalf of Matthew Evans <a class="moz-txt-link-rfc2396E" href="mailto:mattevans123@hotmail.com">
<mattevans123@hotmail.com></a><br>
<b>Sent:</b> Friday, December 9, 2016 9:56 AM<br>
<b>To:</b> Erlang/OTP discussions<br>
<b>Subject:</b> [erlang-questions] R17 - Possible wedged scheduler</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi,</p>
<p><br>
</p>
<p>We just hit a situation where it appeared that 1 scheduler was wedged. Some parts of our application were working, but others appeared to be stuck. I could connect via a cnode application and an escript, but I couldn't connect via the Erlang shell. We have
an escript that does rpc calls, some worked, others (e.g. anything to the code server or tracing failed) failed. </p>
<p><br>
</p>
<p>CPU load was minimal at the time, and heart didn't complain. We only have a single NIF, but this is not called on this hardware variant. We do use CNODE to talk to C applications.</p>
<p><br>
</p>
<p>We are running R17, Intel quad core CPU on Debian.</p>
<p><br>
</p>
<p>This is the first time this has been seen, so the questions are:</p>
<p><br>
</p>
<p>1. Has anyone seen this before?</p>
<p>2. What can we do if we hit this condition in the future to debug?</p>
<p>3. Since heart doesn't detect this can anyone think of any alternative mechanisms?</p>
<p><br>
</p>
<p>Thanks</p>
<p><br>
</p>
<p>Matt</p>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset> <br>
<pre>_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
</body>
</html>