<div dir="ltr">To clarify about our workload, the NIF execution itself is around 1 msec, the data on which is operates represents 10 msec of audio. Apologies if my last message was unclear.<div><br></div><div>Out of convenience, we're using the open source WebRTC project to take advantage of their built in PLC, FEC, Opus, etc. The project is written in C++, so we have integrated with it via NIF. Unfortunately, re-writing using a yielding NIF, or re-writing to Erlang is not as straightforward as we would like. Although I admit it would alleviate our scheduling issues.</div><div><br></div><div>We'll continue our testing with 21.2.3 and report back if there appear to be any other leads.</div><div><br></div><div>Thanks,<br>Jesse</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 22, 2019 at 2:56 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">why do you do it via nif?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_-8063307222018501054gmail_attr">On Wed, Jan 16, 2019 at 6:00 PM Jesse Stimpson <<a href="mailto:jstimpson@republicwireless.com" target="_blank">jstimpson@republicwireless.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">It's possible that during our tests the utilization spike was masked by the collapse issue fixed in the recent PRs. Is there any other analysis I can provide on the utilization spike/sleep behavior we're seeing, or any other debugging or code reading you recommend? As far as I can tell, there's nothing about our workload that would cause periodic behavior like this. The application is slinging RTP audio via udp to remote endpoints at a 20 msec ptime. Each function call for the NIF in question adds 10 msec of audio to the WebRTC buffer.</div><div dir="ltr"><div><br></div><div>As point of corroboration, this user on stackoverflow appears to be having the same or a similar issue: <a href="https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why" target="_blank">https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why</a></div><div><br></div><div>As always, the level of support from the Erlang community is second to none. Thanks to all for your time!</div><div><br></div><div>Jesse</div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_-8063307222018501054gmail-m_4001417713337963958gmail_attr">On Wed, Jan 16, 2019 at 6:35 AM Rickard Green <<a href="mailto:rickard@erlang.org" target="_blank">rickard@erlang.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2019-01-15 23:11, Jesse Stimpson wrote:<br>
> Behavior of the schedulers appears to have the same issue with 2093 patch.<br>
> <br>
> But I did notice something new in the msacc output. There is a very <br>
> brief period, at approx the same time as the normal schedulers usage <br>
> spikes, where all the dirty cpu schedulers have a significant sleep <br>
> time. I've included timestamped excerpts below, starting with the <br>
> increase in dirty cpu sleep, and ending with a "steady state" utilization.<br>
> <br>
<br>
We just released OTP-21.2.3 containing PR-2093.<br>
<br>
I don't think PR-2093 cause the spikes. This change does not affect how <br>
work is moved between normal and dirty schedulers, only prevents the <br>
"loss" of dirty schedulers.<br>
<br>
If a process is scheduled on a dirty scheduler it wont make progress <br>
until it has executed on a dirty scheduler and vice versa (for normal <br>
schedulers). This is the same both before and after PR-2093. Since dirty <br>
schedulers aren't "lost" after PR-2093 progress of such processes will <br>
happen earlier which of course change the behavior, but that is due to <br>
the work load.<br>
<br>
Regards,<br>
Rickard<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-8063307222018501054gmail-m_4001417713337963958gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><table style="font-size:12.8px;border:none;border-collapse:collapse"><tbody><tr style="height:0px"><td style="border-width:0px 1px 0px 0px;border-style:solid;border-color:rgb(0,0,0) rgb(204,204,204) rgb(0,0,0) rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://www.republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/OtQkJgG7cQsvQNGkkWBvl6NRdkaV7AlHeBvQeodb-MDWGY8uq3S6KESFRH6U1dN0-jIXYqrApf_ed4sYYvlmgenTmlizLyUGM9XFjzcS97_3AMyFf-78bzNS2yxkdABXzR3VhwWZul8N0H341w" width="105" height="31" style="border: none;"></span></a></p></td><td style="border-width:0px 0px 0px 1px;border-style:solid;border-color:rgb(0,0,0) rgb(0,0,0) rgb(0,0,0) rgb(204,204,204);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Jesse Stimpson</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Site Reliability Engineering</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">m: </span><span style="font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#1155cc"><a href="tel:(919)%20995-0424" value="+19192971077" style="color:rgb(17,85,204)" target="_blank">9199950424</a></font></span><span style="font-size:13.3333px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="color:rgb(17,85,204);text-decoration:underline;font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank">RepublicWireless.com</a></span></p></td></tr></tbody></table></div></span><div><div dir="ltr"></div></div></div></div></div></div>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><table style="font-size:12.8px;border:none;border-collapse:collapse"><tbody><tr style="height:0px"><td style="border-width:0px 1px 0px 0px;border-style:solid;border-color:rgb(0,0,0) rgb(204,204,204) rgb(0,0,0) rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://www.republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/OtQkJgG7cQsvQNGkkWBvl6NRdkaV7AlHeBvQeodb-MDWGY8uq3S6KESFRH6U1dN0-jIXYqrApf_ed4sYYvlmgenTmlizLyUGM9XFjzcS97_3AMyFf-78bzNS2yxkdABXzR3VhwWZul8N0H341w" width="105" height="31" style="border: none;"></span></a></p></td><td style="border-width:0px 0px 0px 1px;border-style:solid;border-color:rgb(0,0,0) rgb(0,0,0) rgb(0,0,0) rgb(204,204,204);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Jesse Stimpson</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Site Reliability Engineering</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">m: </span><span style="font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#1155cc"><a href="tel:(919)%20995-0424" value="+19192971077" style="color:rgb(17,85,204)" target="_blank">9199950424</a></font></span><span style="font-size:13.3333px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="color:rgb(17,85,204);text-decoration:underline;font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank">RepublicWireless.com</a></span></p></td></tr></tbody></table></div></span><div><div dir="ltr"></div></div></div></div></div></div>