<div dir="ltr">After more testing with 21.2.3, it appears that the behavior we are seeing with respect to dirty schedulers going to sleep occurs only when the dirty scheduler runs on a cpu hyperthread. On our test instance, I disabled hyperthreading and used +SP 50:50, and the problem has gone away. This allows our NIF workload and non-NIF workload an even attribution of CPU resources, which works well for our purposes.<div><br></div><div>It's unclear to us why hyperthreading causes this behavior, but we do wonder if it may have something to do with AWS virtualization. In any case, thanks to everyone for the assistance!</div><div><br></div><div>Best,</div><div>Jesse</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 23, 2019 at 1:14 PM Jesse Stimpson <<a href="mailto:jstimpson@republicwireless.com">jstimpson@republicwireless.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">To clarify about our workload, the NIF execution itself is around 1 msec, the data on which is operates represents 10 msec of audio. Apologies if my last message was unclear.<div><br></div><div>Out of convenience, we're using the open source WebRTC project to take advantage of their built in PLC, FEC, Opus, etc. The project is written in C++, so we have integrated with it via NIF. Unfortunately, re-writing using a yielding NIF, or re-writing to Erlang is not as straightforward as we would like. Although I admit it would alleviate our scheduling issues.</div><div><br></div><div>We'll continue our testing with 21.2.3 and report back if there appear to be any other leads.</div><div><br></div><div>Thanks,<br>Jesse</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_-4473639741031571218gmail_attr">On Tue, Jan 22, 2019 at 2:56 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com" target="_blank">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">why do you do it via nif?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_-4473639741031571218gmail-m_-8063307222018501054gmail_attr">On Wed, Jan 16, 2019 at 6:00 PM Jesse Stimpson <<a href="mailto:jstimpson@republicwireless.com" target="_blank">jstimpson@republicwireless.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">It's possible that during our tests the utilization spike was masked by the collapse issue fixed in the recent PRs. Is there any other analysis I can provide on the utilization spike/sleep behavior we're seeing, or any other debugging or code reading you recommend? As far as I can tell, there's nothing about our workload that would cause periodic behavior like this. The application is slinging RTP audio via udp to remote endpoints at a 20 msec ptime. Each function call for the NIF in question adds 10 msec of audio to the WebRTC buffer.</div><div dir="ltr"><div><br></div><div>As point of corroboration, this user on stackoverflow appears to be having the same or a similar issue: <a href="https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why" target="_blank">https://stackoverflow.com/questions/49563067/erlang-schedulers-just-sleep-why</a></div><div><br></div><div>As always, the level of support from the Erlang community is second to none. Thanks to all for your time!</div><div><br></div><div>Jesse</div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_-4473639741031571218gmail-m_-8063307222018501054gmail-m_4001417713337963958gmail_attr">On Wed, Jan 16, 2019 at 6:35 AM Rickard Green <<a href="mailto:rickard@erlang.org" target="_blank">rickard@erlang.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2019-01-15 23:11, Jesse Stimpson wrote:<br>
> Behavior of the schedulers appears to have the same issue with 2093 patch.<br>
> <br>
> But I did notice something new in the msacc output. There is a very <br>
> brief period, at approx the same time as the normal schedulers usage <br>
> spikes, where all the dirty cpu schedulers have a significant sleep <br>
> time. I've included timestamped excerpts below, starting with the <br>
> increase in dirty cpu sleep, and ending with a "steady state" utilization.<br>
> <br>
<br>
We just released OTP-21.2.3 containing PR-2093.<br>
<br>
I don't think PR-2093 cause the spikes. This change does not affect how <br>
work is moved between normal and dirty schedulers, only prevents the <br>
"loss" of dirty schedulers.<br>
<br>
If a process is scheduled on a dirty scheduler it wont make progress <br>
until it has executed on a dirty scheduler and vice versa (for normal <br>
schedulers). This is the same both before and after PR-2093. Since dirty <br>
schedulers aren't "lost" after PR-2093 progress of such processes will <br>
happen earlier which of course change the behavior, but that is due to <br>
the work load.<br>
<br>
Regards,<br>
Rickard<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-4473639741031571218gmail-m_-8063307222018501054gmail-m_4001417713337963958gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><table style="font-size:12.8px;border:none;border-collapse:collapse"><tbody><tr style="height:0px"><td style="border-width:0px 1px 0px 0px;border-style:solid;border-color:rgb(0,0,0) rgb(204,204,204) rgb(0,0,0) rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://www.republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/OtQkJgG7cQsvQNGkkWBvl6NRdkaV7AlHeBvQeodb-MDWGY8uq3S6KESFRH6U1dN0-jIXYqrApf_ed4sYYvlmgenTmlizLyUGM9XFjzcS97_3AMyFf-78bzNS2yxkdABXzR3VhwWZul8N0H341w" width="105" height="31" style="border: none;"></span></a></p></td><td style="border-width:0px 0px 0px 1px;border-style:solid;border-color:rgb(0,0,0) rgb(0,0,0) rgb(0,0,0) rgb(204,204,204);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Jesse Stimpson</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Site Reliability Engineering</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">m: </span><span style="font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#1155cc"><a href="tel:(919)%20995-0424" value="+19192971077" style="color:rgb(17,85,204)" target="_blank">9199950424</a></font></span><span style="font-size:13.3333px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="color:rgb(17,85,204);text-decoration:underline;font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank">RepublicWireless.com</a></span></p></td></tr></tbody></table></div></span><div><div dir="ltr"></div></div></div></div></div></div>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-4473639741031571218gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><table style="font-size:12.8px;border:none;border-collapse:collapse"><tbody><tr style="height:0px"><td style="border-width:0px 1px 0px 0px;border-style:solid;border-color:rgb(0,0,0) rgb(204,204,204) rgb(0,0,0) rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://www.republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/OtQkJgG7cQsvQNGkkWBvl6NRdkaV7AlHeBvQeodb-MDWGY8uq3S6KESFRH6U1dN0-jIXYqrApf_ed4sYYvlmgenTmlizLyUGM9XFjzcS97_3AMyFf-78bzNS2yxkdABXzR3VhwWZul8N0H341w" width="105" height="31" style="border: none;"></span></a></p></td><td style="border-width:0px 0px 0px 1px;border-style:solid;border-color:rgb(0,0,0) rgb(0,0,0) rgb(0,0,0) rgb(204,204,204);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Jesse Stimpson</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Site Reliability Engineering</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">m: </span><span style="font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#1155cc"><a href="tel:(919)%20995-0424" value="+19192971077" style="color:rgb(17,85,204)" target="_blank">9199950424</a></font></span><span style="font-size:13.3333px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="color:rgb(17,85,204);text-decoration:underline;font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank">RepublicWireless.com</a></span></p></td></tr></tbody></table></div></span><div><div dir="ltr"></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><table style="font-size:12.8px;border:none;border-collapse:collapse"><tbody><tr style="height:0px"><td style="border-width:0px 1px 0px 0px;border-style:solid;border-color:rgb(0,0,0) rgb(204,204,204) rgb(0,0,0) rgb(0,0,0);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://www.republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh6.googleusercontent.com/OtQkJgG7cQsvQNGkkWBvl6NRdkaV7AlHeBvQeodb-MDWGY8uq3S6KESFRH6U1dN0-jIXYqrApf_ed4sYYvlmgenTmlizLyUGM9XFjzcS97_3AMyFf-78bzNS2yxkdABXzR3VhwWZul8N0H341w" width="105" height="31" style="border: none;"></span></a></p></td><td style="border-width:0px 0px 0px 1px;border-style:solid;border-color:rgb(0,0,0) rgb(0,0,0) rgb(0,0,0) rgb(204,204,204);vertical-align:top;padding:0px"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Jesse Stimpson</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Site Reliability Engineering</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;margin-left:9.75pt"><span style="font-size:13.3333px;font-family:Arial;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">m: </span><span style="font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font color="#1155cc"><a href="tel:(919)%20995-0424" value="+19192971077" style="color:rgb(17,85,204)" target="_blank">9199950424</a></font></span><span style="font-size:13.3333px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="color:rgb(17,85,204);text-decoration:underline;font-size:13.3333px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://republicwireless.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank">RepublicWireless.com</a></span></p></td></tr></tbody></table></div></span><div><div dir="ltr"></div></div></div></div></div></div>