[erlang-questions] How to test multi-pollset?

[pablo platt]

On Fri, Feb 23, 2018 at 5:10 PM, Lukas Larsson <lukas@REDACTED> wrote:

> Hello,
>
> On Fri, Feb 23, 2018 at 3:15 PM, pablo platt <pablo.platt@REDACTED>
> wrote:
>
>> Master with multi poll-sets is running on my dev machine without errors
>> so far.
>>
>
> great!
>
>
>>
>> What's the difference between polling threads and poll-sets (+IOt  and
>> +IOp)?
>>
>
> Its explained in this presentation: http://www.erlang-factory.com/euc2017/
> kenneth-lundin
>
> Polling threads can be added when its load (as measured by msacc)
> increases. In my stress tests I've seen a small decrease in latency on
> large machines when adding a second thread.
> The number of poll-sets can be increased to deal with scalability issues
> in the underlying libc/kernel implementation. Modern operating systems
> rarely have any problems with this, so I'm not sure how useful changing
> this value actually is.
>
>
>> How do I know if I should increase the number of polling threads or
>> poll-sets?
>>
>
> The best way is probably through experimentation. Measure your
> applications latency/throughput and then play with the settings and see if
> it changes.
>
>
>> I'm using a 8 or 16 vCPUs machine (vCPU =  hyper-threads) running Ubuntu
>> 16.04.
>>
>> I've measured with msacc as the docs [1] recommends on 1 vCPU machine.
>> Each gen_udp receiving 100 UDP packets per second increase the 'poll' row
>> load by about 0.05%.
>> Adding several gen_udp that send 100 UDP packets per second almost
>> doesn't affect the load.
>> Is it expected that gen_udp receiving packets has high load but gen_udp
>> sending packets very low load?
>>
>
> yes, especially if you use active mode when receiving. Sending UDP
> shouldn't really go via the poll implementation at all as if the kernel
> buffer is full it will just discard the packet.
>

I'm using gen_udp with {active, once} and {recbuf, 16384}.
So if I have 5 gen_udp receiving packets and 500 gen_udp sending packets,
I'll probably won't see a big performance difference from multi poll-sets.


>
>
>>
>> How can I compare master with multiple poll-sets with erlang/otp 20?
>>
>
> Run your application and measure using external tools the difference in
> throughput/latency.
>

I will. Thanks.


>
> Lukas
>
>
>> msacc on otp 20 doesn't have stats about poll.
>>
>> msacc docs [2] don't show example output of poll load. Maybe not that
>> important.
>>
>> [1] https://github.com/erlang/otp/blob/master/erts/doc/src/erl.xml
>> [2] https://github.com/erlang/otp/blob/master/lib/runtime_tools/
>> doc/src/msacc.xml
>>
>>
>> On Mon, Jan 29, 2018 at 11:19 AM, pablo platt <pablo.platt@REDACTED>
>> wrote:
>>
>>> It's great to see all the hard work invested in performance in master.
>>> Thanks.
>>>
>>> On Mon, Jan 29, 2018 at 10:49 AM, Lukas Larsson <lukas@REDACTED>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> On Mon, Jan 29, 2018 at 9:22 AM, pablo platt <pablo.platt@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> What is the expected effect of the multi-pollset PR [1] on a UDP
>>>>> socket on the sender/receiver side?
>>>>> My use case is a media server with several broadcaster and many
>>>>> viewers.
>>>>> Each stream use 1Mbps (aprox 100 * 1500 bytes packets per second).
>>>>> Should I expect improvement when gen_udp is sending packets, receiving
>>>>> packets or both?
>>>>>
>>>>
>>>> Yes, I believe that you will see an improvement. It depends on what
>>>> type of HW that you are running on, typically the more logical cpu's you
>>>> have the more gain you will get from the improvements in I/O polling[1].
>>>> Also the exact usage pattern matters.
>>>>
>>>>
>>>>> Is it reasonable to pick a point in master and use it on a production
>>>>> system after testing?
>>>>>
>>>>
>>>> I would take the latest tip of master and test that thoroughly for you
>>>> application. The things that we merge into master have gone through all our
>>>> testing before it is merged, so it is as stable as the maint branch.
>>>> However we make a lot more changes in master than in maint, so because of
>>>> that there will be a greater chance of some bug slipping through.
>>>>
>>>> If you do decide to give the improved I/O polling implementation a go,
>>>> please do come back with any negative or positive findings that you get!
>>>>
>>>> Lukas
>>>>
>>>> [1]: The largest change in the PR is not actually the ability to use
>>>> multiple pollsets, but that the polling has been lifted out to be done by
>>>> dedicated threads.
>>>>
>>>
>>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180223/62f4fc94/attachment.htm>