[erlang-patches] Pollset per scheduler and bind port to scheduler

[Wei Cao]

2012/7/11 Lukas Larsson <lukas@REDACTED>:
> Hi,
>
> The reason I'm skeptical about anything which binds processes/ports to
> scheduler is that it feels like a temporary solution and would much rather
> do a proper solution where the scheduler takes care of these things for you.
> But as I said, internally we need to talk this over when it is not in the
> middle of summer vacation.
>
> I did some benchmarking using ab and found basically the same figures as
> you. The below is with keep-alive and the values are requests per second:
>
>                    not-bound        bound
>
> R15B01                 44k        37k
>
> master                  44k        35k
>
> master+mp          48k        49k
>
> master+mp+pb    49k        55k
>
> [mp]: multi-poll patch
> [pb]: port bind patch
> [bound]: Used {scheduler,I} to spread load
>
> Unfortunately I also found that when doing the non-keep alive benchmark the
> performance is seriously degraded.
>
> R15B01 not-bound                  8255
> master+mp+pb not-bound    7668
> master+mp+pb bound           5765
>

I found why performance degrates in non-keep alive benchmark. It's
caused by waking schedulers up to do aux works too frequently.

After applying pollset per scheduler patch, each scheduler tends to
wait in poll operation if there is no heavy load. So if an aux work
arrives at this time, erts_check_io_interrupt() will be called and
write a byte to a pipe fd to wake up the scheduler from poll
operation.

Unfortunately, there're quite a lot of aux works in Erlang VM, such as
delayed dealloc, so schedulers're frequently woken up from poll to
process these tasks, say, deallocate a memory block, and go into poll
again.

In a non-keep alive benchmark with 15k QPS, schedulers're woken up
about 300k times per second (I hacked the code adding a atomic counter
to record it), it's really time cosuming.

After commenting out the code which wakes up scheduler to process aux
work, the QPS increases to 20k, which proves above suspect.

I think it's really CPU time wasting to wake up scheduler from polling
to deallocate some memory blocks etc, may be it's more suitable to
wake schedulers up periodically (say, 1 millisecond) to process them,
or doing aux tasks in the aux_thread only. Any ideas to fix this?


> I did some gprof runs but could not find anything obvious that is going
> wrong.
>
> Lukas
>
>
> On 11/07/12 04:21, Wei Cao wrote:
>>
>> I added a macro to conditional compile the patch because I think it
>> can be more selectable, I can remove the macro, fix the compilation
>> error and test on mingw platform in later version.
>>
>> how about provide another BIF named port_flag (like process_flag) to
>> let user bind port to a given scheduler?
>>
>>
>



-- 

Best,

Wei Cao