<!DOCTYPE html>
<html>
<head>
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
</head>
<body><div>On Thu, 28 Jun 2018, at 22:35, Maxim Fedorov wrote:<br></div>
<div>> I’m trying to debug some weird condition when any misc system task hangs.<br></div>
<div>> It seems to affect OTP 20 (but not 16) on FreeBSD 10.3 and 11.<br></div>
<div>><br></div>
<div>> It is a rare problem happening after 5-7 days under some load (~40% cpu<br></div>
<div>> average on a 48 cores server).<br></div>
<div>><br></div>
<div>> There is also a problem with erlang:statistics(runtime), affected by<br></div>
<div>> this bug in FreeBSD kernel:<br></div>
<div>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227689 (so<br></div>
<div>> statistics:runtime() always returns the same value), however I doubt it<br></div>
<div>> can affect anything.<br></div>
<div>><br></div>
<div>> What happens: there are several calls, e.g.<br></div>
<div>> erlang:statistics(garbage_collection), ets:all(),<br></div>
<div>> erts_internal:system_check() and few more. All of them do<br></div>
<div>> erts_schedule_misc_aux_work. A misc aux work item is put into every<br></div>
<div>> scheduler queue, and it seems that all of them except one respond. VM is<br></div>
<div>> still working, all other processes are fine, but the one that did the<br></div>
<div>> call is waiting in erlang:gc_info/2 (or another corresponding function),<br></div>
<div>> with counter equals to 1. Since there is no timeout in receive<br></div>
<div>> statement, it waits forever.<br></div>
<div>><br></div>
<div>> How do I debug this?<br></div>
<div><br></div>
<div>I get that your load in production is high, but will a targeted dtrace probe be lightweight enough?<br></div>
<div><br></div>
<div>Also wrt Anthony’s comments are you building with clang or gcc? I’m not clear if that’s relevant<br></div>
<div><br></div>
<div>A+<br></div>
<div>Dave</div>
<div id="sig20903217"><div class="signature"><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
</div>
<div><br></div>
</body>
</html>