[erlang-patches] Pollset per scheduler and bind port to scheduler

[Lukas Larsson]

Hello!

Sorry for the lack of replies, I've been away from the office the last 
two weeks.

We have done some changes in the master which could affect the 
performance of the work you did: 
https://github.com/erlang/otp/commit/8457bb3aeb335733d22ab6d517fe173cd90b4f55
Could you check and see if these changes affect any of your benchmarks?

It would indeed be better to wake interrupt the poll less often if 
possible. However measuring time is generally quite time consuming, so 
we want to avoid it as much as possible, maybe use a scheduler specific 
counter? Its a fine balance though as we do not want to delay dealloc of 
memory for too long.

Lukas


On 24/07/12 11:57, Wei Cao wrote:
> 2012/7/11 Lukas Larsson <lukas@REDACTED>:
>> Hi,
>>
>> The reason I'm skeptical about anything which binds processes/ports to
>> scheduler is that it feels like a temporary solution and would much rather
>> do a proper solution where the scheduler takes care of these things for you.
>> But as I said, internally we need to talk this over when it is not in the
>> middle of summer vacation.
>>
>> I did some benchmarking using ab and found basically the same figures as
>> you. The below is with keep-alive and the values are requests per second:
>>
>>                     not-bound        bound
>>
>> R15B01                 44k        37k
>>
>> master                  44k        35k
>>
>> master+mp          48k        49k
>>
>> master+mp+pb    49k        55k
>>
>> [mp]: multi-poll patch
>> [pb]: port bind patch
>> [bound]: Used {scheduler,I} to spread load
>>
>> Unfortunately I also found that when doing the non-keep alive benchmark the
>> performance is seriously degraded.
>>
>> R15B01 not-bound                  8255
>> master+mp+pb not-bound    7668
>> master+mp+pb bound           5765
>>
> I found why performance degrates in non-keep alive benchmark. It's
> caused by waking schedulers up to do aux works too frequently.
>
> After applying pollset per scheduler patch, each scheduler tends to
> wait in poll operation if there is no heavy load. So if an aux work
> arrives at this time, erts_check_io_interrupt() will be called and
> write a byte to a pipe fd to wake up the scheduler from poll
> operation.
>
> Unfortunately, there're quite a lot of aux works in Erlang VM, such as
> delayed dealloc, so schedulers're frequently woken up from poll to
> process these tasks, say, deallocate a memory block, and go into poll
> again.
>
> In a non-keep alive benchmark with 15k QPS, schedulers're woken up
> about 300k times per second (I hacked the code adding a atomic counter
> to record it), it's really time cosuming.
>
> After commenting out the code which wakes up scheduler to process aux
> work, the QPS increases to 20k, which proves above suspect.
>
> I think it's really CPU time wasting to wake up scheduler from polling
> to deallocate some memory blocks etc, may be it's more suitable to
> wake schedulers up periodically (say, 1 millisecond) to process them,
> or doing aux tasks in the aux_thread only. Any ideas to fix this?
>
>
>> I did some gprof runs but could not find anything obvious that is going
>> wrong.
>>
>> Lukas
>>
>>
>> On 11/07/12 04:21, Wei Cao wrote:
>>> I added a macro to conditional compile the patch because I think it
>>> can be more selectable, I can remove the macro, fix the compilation
>>> error and test on mingw platform in later version.
>>>
>>> how about provide another BIF named port_flag (like process_flag) to
>>> let user bind port to a given scheduler?
>>>
>>>
>
>