<div dir="ltr"><div><div><div><div><div><div>Hello,<br><br></div><div>Updating with my test.<br><br></div>I've been using master in production for two days on 4 servers.<br></div>Each server has 4 vCPUs.<br>Each server handles about 10K UDP packets per second. Approx 1K incoming packets per second and 9K outgoing packets per second.<br><br></div>I didn't see any issues. Seems to work fine.<br></div>CPU load in OTP Master 21 increased compared to OTP 20 from 58% to 68%. Does this make sense?<br></div>This is a real production system that does other things in addition to just sending UDP packets but the OTP version is the only change.<br><br></div>Thanks<br><div><div><div><div><div><br><div><br><br></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 23, 2018 at 5:29 PM, pablo platt <span dir="ltr"><<a href="mailto:pablo.platt@gmail.com" target="_blank">pablo.platt@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Fri, Feb 23, 2018 at 5:10 PM, Lukas Larsson <span dir="ltr"><<a href="mailto:lukas@erlang.org" target="_blank">lukas@erlang.org</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<br><div class="gmail_extra"><br><span class=""><div class="gmail_quote"><span class="m_-5565952633771900164gmail-">On Fri, Feb 23, 2018 at 3:15 PM, pablo platt <span dir="ltr"><<a href="mailto:pablo.platt@gmail.com" target="_blank">pablo.platt@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Master with multi poll-sets is running on my dev machine without errors so far.<br></div></div></div></div></div></div></blockquote><div><br></div></span><div>great!</div><span class="m_-5565952633771900164gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><br>What's the difference between
polling threads and poll-sets (<code>+IOt</code>
and
<code>+IOp</code>)?<br></div></div></div></div></div></div></blockquote><div><br></div></span><div>Its explained in this presentation: <a href="http://www.erlang-factory.com/euc2017/kenneth-lundin" target="_blank">http://www.erlan<wbr>g-factory.com/euc2017/kenneth-<wbr>lundin</a></div><div><br></div><div>Polling threads can be added when its load (as measured by msacc) increases. In my stress tests I've seen a small decrease in latency on large machines when adding a second thread.</div><div>The number of poll-sets can be increased to deal with scalability issues in the underlying libc/kernel implementation. Modern operating systems rarely have any problems with this, so I'm not sure how useful changing this value actually is.</div><span class="m_-5565952633771900164gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div></div><div>How do I know if I should increase the number of polling threads or poll-sets?</div></div></div></div></div></div></blockquote><div><br></div></span><div>The best way is probably through experimentation. Measure your applications latency/throughput and then play with the settings and see if it changes.</div><span class="m_-5565952633771900164gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div></div><div>I'm using a 8 or 16 vCPUs machine (vCPU = hyper-threads) running Ubuntu 16.04.<br><br></div>I've measured with <code>msacc</code> as the docs [1] recommends on 1 vCPU machine.<br></div>Each gen_udp receiving 100 UDP packets per second increase the 'poll' row load by about 0.05%.<br></div>Adding several gen_udp that send 100 UDP packets per second almost doesn't affect the load.<br></div><div>Is it expected that gen_udp receiving packets has high load but gen_udp sending packets very low load?<br></div></div></div></blockquote><div><br></div></span><div>yes, especially if you use active mode when receiving. Sending UDP shouldn't really go via the poll implementation at all as if the kernel buffer is full it will just discard the packet.</div></div></span></div></div></blockquote><div><br></div><div>I'm using gen_udp with {active, once} and {recbuf, 16384}.<br></div><div>So if I have 5 gen_udp receiving packets and 500 gen_udp sending packets, I'll probably won't see a big performance difference from multi poll-sets.<br></div><span class=""><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5565952633771900164gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div></div><div><br></div>How can I compare master with multiple poll-sets with erlang/otp 20?<br></div></div></blockquote><div><br></div></span><div>Run your application and measure using external tools the difference in throughput/latency.</div></div></div></div></blockquote><div><br></div></span><div>I will. Thanks.<br> <br></div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Lukas</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_-5565952633771900164gmail-"><div dir="ltr"><div></div>
<code>msacc</code>
on otp 20 doesn't have stats about poll.<br><br><div><div><div>msacc docs [2] don't show example output of poll load. Maybe not that important.<br><div><br>[1] <a href="https://github.com/erlang/otp/blob/master/erts/doc/src/erl.xml" target="_blank">https://github.com/erlang/otp/<wbr>blob/master/erts/doc/src/erl.x<wbr>ml</a><br>[2]
<code><a href="https://github.com/erlang/otp/blob/master/lib/runtime_tools/doc/src/msacc.xml" target="_blank">https://github.com/erlang/otp/<wbr>blob/master/lib/runtime_tools/<wbr>doc/src/msacc.xml</a></code>
<br><code></code> <br></div></div></div></div></div><div class="m_-5565952633771900164gmail-m_-7638565484233333139gmail-HOEnZb"><div class="m_-5565952633771900164gmail-m_-7638565484233333139gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 11:19 AM, pablo platt <span dir="ltr"><<a href="mailto:pablo.platt@gmail.com" target="_blank">pablo.platt@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It's great to see all the hard work invested in performance in master.<br>Thanks.<br></div><div class="m_-5565952633771900164gmail-m_-7638565484233333139gmail-m_-5202283986903528267HOEnZb"><div class="m_-5565952633771900164gmail-m_-7638565484233333139gmail-m_-5202283986903528267h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 10:49 AM, Lukas Larsson <span dir="ltr"><<a href="mailto:lukas@erlang.org" target="_blank">lukas@erlang.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Jan 29, 2018 at 9:22 AM, pablo platt <span dir="ltr"><<a href="mailto:pablo.platt@gmail.com" target="_blank">pablo.platt@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,<br><br></div><div>What is the expected effect of the multi-pollset PR [1] on a UDP socket on the sender/receiver side?<br></div><div>My use case is a media server with several broadcaster and many viewers.<br></div><div>Each stream use 1Mbps (aprox 100 * 1500 bytes packets per second).<br></div><div>Should I expect improvement when gen_udp is sending packets, receiving packets or both?<br></div></div></blockquote><div><br></div></span><div>Yes, I believe that you will see an improvement. It depends on what type of HW that you are running on, typically the more logical cpu's you have the more gain you will get from the improvements in I/O polling[1]. Also the exact usage pattern matters.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Is it reasonable to pick a point in master and use it on a production system after testing?<br></div></div></blockquote><div><br></div></span><div>I would take the latest tip of master and test that thoroughly for you application. The things that we merge into master have gone through all our testing before it is merged, so it is as stable as the maint branch. However we make a lot more changes in master than in maint, so because of that there will be a greater chance of some bug slipping through.</div><div><br></div><div>If you do decide to give the improved I/O polling implementation a go, please do come back with any negative or positive findings that you get!</div><div><br></div><div>Lukas</div><div><br></div><div>[1]: The largest change in the PR is not actually the ability to use multiple pollsets, but that the polling has been lifted out to be done by dedicated threads.</div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br></span>______________________________<wbr>_________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/list<wbr>info/erlang-questions</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>