[erlang-questions] How to test multi-pollset?

[Lukas Larsson]

Hello,

On Fri, Feb 23, 2018 at 3:15 PM, pablo platt <pablo.platt@REDACTED> wrote:

> Master with multi poll-sets is running on my dev machine without errors so
> far.
>

great!


>
> What's the difference between polling threads and poll-sets (+IOt  and
> +IOp)?
>

Its explained in this presentation:
http://www.erlang-factory.com/euc2017/kenneth-lundin

Polling threads can be added when its load (as measured by msacc)
increases. In my stress tests I've seen a small decrease in latency on
large machines when adding a second thread.
The number of poll-sets can be increased to deal with scalability issues in
the underlying libc/kernel implementation. Modern operating systems rarely
have any problems with this, so I'm not sure how useful changing this value
actually is.


> How do I know if I should increase the number of polling threads or
> poll-sets?
>

The best way is probably through experimentation. Measure your applications
latency/throughput and then play with the settings and see if it changes.


> I'm using a 8 or 16 vCPUs machine (vCPU =  hyper-threads) running Ubuntu
> 16.04.
>
> I've measured with msacc as the docs [1] recommends on 1 vCPU machine.
> Each gen_udp receiving 100 UDP packets per second increase the 'poll' row
> load by about 0.05%.
> Adding several gen_udp that send 100 UDP packets per second almost doesn't
> affect the load.
> Is it expected that gen_udp receiving packets has high load but gen_udp
> sending packets very low load?
>

yes, especially if you use active mode when receiving. Sending UDP
shouldn't really go via the poll implementation at all as if the kernel
buffer is full it will just discard the packet.


>
> How can I compare master with multiple poll-sets with erlang/otp 20?
>

Run your application and measure using external tools the difference in
throughput/latency.

Lukas


> msacc on otp 20 doesn't have stats about poll.
>
> msacc docs [2] don't show example output of poll load. Maybe not that
> important.
>
> [1] https://github.com/erlang/otp/blob/master/erts/doc/src/erl.xml
> [2] https://github.com/erlang/otp/blob/master/lib/runtime_tools/
> doc/src/msacc.xml
>
>
> On Mon, Jan 29, 2018 at 11:19 AM, pablo platt <pablo.platt@REDACTED>
> wrote:
>
>> It's great to see all the hard work invested in performance in master.
>> Thanks.
>>
>> On Mon, Jan 29, 2018 at 10:49 AM, Lukas Larsson <lukas@REDACTED> wrote:
>>
>>> Hello,
>>>
>>> On Mon, Jan 29, 2018 at 9:22 AM, pablo platt <pablo.platt@REDACTED>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> What is the expected effect of the multi-pollset PR [1] on a UDP socket
>>>> on the sender/receiver side?
>>>> My use case is a media server with several broadcaster and many viewers.
>>>> Each stream use 1Mbps (aprox 100 * 1500 bytes packets per second).
>>>> Should I expect improvement when gen_udp is sending packets, receiving
>>>> packets or both?
>>>>
>>>
>>> Yes, I believe that you will see an improvement. It depends on what type
>>> of HW that you are running on, typically the more logical cpu's you have
>>> the more gain you will get from the improvements in I/O polling[1]. Also
>>> the exact usage pattern matters.
>>>
>>>
>>>> Is it reasonable to pick a point in master and use it on a production
>>>> system after testing?
>>>>
>>>
>>> I would take the latest tip of master and test that thoroughly for you
>>> application. The things that we merge into master have gone through all our
>>> testing before it is merged, so it is as stable as the maint branch.
>>> However we make a lot more changes in master than in maint, so because of
>>> that there will be a greater chance of some bug slipping through.
>>>
>>> If you do decide to give the improved I/O polling implementation a go,
>>> please do come back with any negative or positive findings that you get!
>>>
>>> Lukas
>>>
>>> [1]: The largest change in the PR is not actually the ability to use
>>> multiple pollsets, but that the polling has been lifted out to be done by
>>> dedicated threads.
>>>
>>
>>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180223/ab3e2ca1/attachment.htm>