[erlang-questions] How to test multi-pollset?

[pablo platt]

Hello,

Updating with my test.

I've been using master in production for two days on 4 servers.
Each server has 4 vCPUs.
Each server handles about 10K UDP packets per second. Approx 1K incoming
packets per second and 9K outgoing packets per second.

I didn't see any issues. Seems to work fine.
CPU load in OTP Master 21 increased compared to OTP 20 from 58% to 68%.
Does this make sense?
This is a real production system that does other things in addition to just
sending UDP packets but the OTP version is the only change.

Thanks




On Fri, Feb 23, 2018 at 5:29 PM, pablo platt <pablo.platt@REDACTED> wrote:

>
>
> On Fri, Feb 23, 2018 at 5:10 PM, Lukas Larsson <lukas@REDACTED> wrote:
>
>> Hello,
>>
>> On Fri, Feb 23, 2018 at 3:15 PM, pablo platt <pablo.platt@REDACTED>
>> wrote:
>>
>>> Master with multi poll-sets is running on my dev machine without errors
>>> so far.
>>>
>>
>> great!
>>
>>
>>>
>>> What's the difference between polling threads and poll-sets (+IOt  and
>>> +IOp)?
>>>
>>
>> Its explained in this presentation: http://www.erlan
>> g-factory.com/euc2017/kenneth-lundin
>>
>> Polling threads can be added when its load (as measured by msacc)
>> increases. In my stress tests I've seen a small decrease in latency on
>> large machines when adding a second thread.
>> The number of poll-sets can be increased to deal with scalability issues
>> in the underlying libc/kernel implementation. Modern operating systems
>> rarely have any problems with this, so I'm not sure how useful changing
>> this value actually is.
>>
>>
>>> How do I know if I should increase the number of polling threads or
>>> poll-sets?
>>>
>>
>> The best way is probably through experimentation. Measure your
>> applications latency/throughput and then play with the settings and see if
>> it changes.
>>
>>
>>> I'm using a 8 or 16 vCPUs machine (vCPU =  hyper-threads) running Ubuntu
>>> 16.04.
>>>
>>> I've measured with msacc as the docs [1] recommends on 1 vCPU machine.
>>> Each gen_udp receiving 100 UDP packets per second increase the 'poll'
>>> row load by about 0.05%.
>>> Adding several gen_udp that send 100 UDP packets per second almost
>>> doesn't affect the load.
>>> Is it expected that gen_udp receiving packets has high load but gen_udp
>>> sending packets very low load?
>>>
>>
>> yes, especially if you use active mode when receiving. Sending UDP
>> shouldn't really go via the poll implementation at all as if the kernel
>> buffer is full it will just discard the packet.
>>
>
> I'm using gen_udp with {active, once} and {recbuf, 16384}.
> So if I have 5 gen_udp receiving packets and 500 gen_udp sending packets,
> I'll probably won't see a big performance difference from multi poll-sets.
>
>
>>
>>
>>>
>>> How can I compare master with multiple poll-sets with erlang/otp 20?
>>>
>>
>> Run your application and measure using external tools the difference in
>> throughput/latency.
>>
>
> I will. Thanks.
>
>
>>
>> Lukas
>>
>>
>>> msacc on otp 20 doesn't have stats about poll.
>>>
>>> msacc docs [2] don't show example output of poll load. Maybe not that
>>> important.
>>>
>>> [1] https://github.com/erlang/otp/blob/master/erts/doc/src/erl.xml
>>> [2] https://github.com/erlang/otp/blob/master/lib/runtime_tools/
>>> doc/src/msacc.xml
>>>
>>>
>>> On Mon, Jan 29, 2018 at 11:19 AM, pablo platt <pablo.platt@REDACTED>
>>> wrote:
>>>
>>>> It's great to see all the hard work invested in performance in master.
>>>> Thanks.
>>>>
>>>> On Mon, Jan 29, 2018 at 10:49 AM, Lukas Larsson <lukas@REDACTED>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> On Mon, Jan 29, 2018 at 9:22 AM, pablo platt <pablo.platt@REDACTED>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What is the expected effect of the multi-pollset PR [1] on a UDP
>>>>>> socket on the sender/receiver side?
>>>>>> My use case is a media server with several broadcaster and many
>>>>>> viewers.
>>>>>> Each stream use 1Mbps (aprox 100 * 1500 bytes packets per second).
>>>>>> Should I expect improvement when gen_udp is sending packets,
>>>>>> receiving packets or both?
>>>>>>
>>>>>
>>>>> Yes, I believe that you will see an improvement. It depends on what
>>>>> type of HW that you are running on, typically the more logical cpu's you
>>>>> have the more gain you will get from the improvements in I/O polling[1].
>>>>> Also the exact usage pattern matters.
>>>>>
>>>>>
>>>>>> Is it reasonable to pick a point in master and use it on a production
>>>>>> system after testing?
>>>>>>
>>>>>
>>>>> I would take the latest tip of master and test that thoroughly for you
>>>>> application. The things that we merge into master have gone through all our
>>>>> testing before it is merged, so it is as stable as the maint branch.
>>>>> However we make a lot more changes in master than in maint, so because of
>>>>> that there will be a greater chance of some bug slipping through.
>>>>>
>>>>> If you do decide to give the improved I/O polling implementation a go,
>>>>> please do come back with any negative or positive findings that you get!
>>>>>
>>>>> Lukas
>>>>>
>>>>> [1]: The largest change in the PR is not actually the ability to use
>>>>> multiple pollsets, but that the polling has been lifted out to be done by
>>>>> dedicated threads.
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180308/df8ca5ef/attachment.htm>