[erlang-patches] Pollset per scheduler and bind port to scheduler

[Lukas Larsson]

Hello!

The multiple-pollset branch seems to be running without any testcase 
failures in our nightly builds now. We will have to review the code in 
detail and then get back to you, unfortunately that could take a while 
as it is in the middle of vacations times and we are also closing in on 
the next maintenance release.

Regarding binding ports to schedulers:

We do not want to include the functionality the way it is now. It builds 
on a mechanism which really should only be used for testing and can 
create unforeseeable problems with process/port scheduling and scheduler 
load balancing.

If the functionality was changed so that a process_flag could be set on 
a process so that all ports created by that process are bound to the 
process (and migrate with the process), you could still achieve the same 
results but through a functionality which is more general. Depending on 
what changes have to be made to the erlang vm, we might accept such a 
patch. If it works we might even document and support the functionality.

Lukas

On 19/07/12 04:48, Wei Cao wrote:
> Thanks for pointing it out, fixed and pushed to pollset_per_scheduler
> branch at git://github.com/weicao/otp.git
>
> 2012/7/18 Lukas Larsson <lukas@REDACTED>:
>> fyi the patch has introduced three new gcc warnings:
>>
>> erl_port_task.c:230, gcc4,
>>
>> control reaches end of non-void function [-Wreturn-type]
>>
>> erl_check_io.c:992, gcc4,
>>
>> ‘inps’ may be used uninitialized in this function [-Wuninitialized]
>>
>> erl_check_io.c:998, gcc4,
>>
>> ‘outps’ may be used uninitialized in this function [-Wuninitialized]
>>
>> Lukas
>>
>>
>> On 18/07/12 11:28, Wei Cao wrote:
>>> 2012/7/17 Lukas Larsson <lukas@REDACTED>:
>>>> Hi!
>>>>
>>>> After fixing the patch to compile on windows[1] I put it in our daily
>>>> builds
>>>> and a couple of issues came up. I did not include 'move erts_deliver_time
>>>> out of erl_check_io' commit as it deadlocks the non-smp emulator.
>>> This problem is also fixed by testing macro ERTS_SMP to tell whether
>>> it's a non-smp emulator, now both smp and non-smp emulators work.
>>>
>>>> When running a debug test build on Linux we got the following assertion:
>>>> Assertion failed: 0 <= (ix) && (ix) < erts_no_pollsets in
>>>> sys/common/erl_check_io.c, line 1909
>>>>
>>>> On OS X Lion we got the following when compiling gs:
>>>> erl -pa ../ebin -s gs_make -s erlang halt -noshell
>>>> ../include/internal/ethr_mutex.h:655: Fatal error in ethr_mutex_lock():
>>>> Invalid argument (22)
>>>> make[3]: *** [gstk_generic.hrl] Abort trap: 6
>>>> make[2]: *** [opt] Error 2
>>>> make[1]: *** [opt] Error 2
>>>> make: *** [libs] Error 2
>>>>
>>>> I'll remove the branch and see if the same problems appear again tomorrow
>>>> (unfortunately we do not have enough machines to let your branch run
>>>> alone,
>>>> so it might be some other branch causing this). Let me know if you need
>>>> any
>>>> help tracking down these issues.
>>>>
>>>> Lukas
>>>>
>>>> [1]: https://github.com/garazdawi/otp/tree/wc/pollset_per_scheduler
>>>>
>>>>
>>>> On 11/07/12 17:34, Wei Cao wrote:
>>>>
>>>> In non keep-alive cases, all new connections 're accepted by the
>>>> Erlang port which listens on the TCP port, and how frequently/fast the
>>>> port be scheduled to run limits the QPS. (requests per second), so
>>>> this port can be regarded as bottleneck of non keep-alive
>>>> applications.
>>>>
>>>> So I guess performance degradation observed is caused by the listener
>>>> port not be scheduled frequent or fast enough, I'll look into this
>>>> problem tomorrow, now is at night in China, :-)
>>>>
>>>> BTW, I found this patch should be compiled like this today,
>>>> ./configure CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3
>>>> -fomit-frame-pointer"
>>>> otherwise compiler optimization is disabled.
>>>>
>>>> Regarding binding processes/ports to scheduler, I admit it's really a
>>>> temporary solution to bind port to the same scheduler as its owner
>>>> process like the pb patch did, and I suggest it's better to add a
>>>> additional BIF like erlang:process_flag, to allow user explicitly bind
>>>> port to a given scheduler, if it benefits.
>>>>
>>>>
>>>> 2012/7/11 Lukas Larsson <lukas@REDACTED>:
>>>>
>>>> Hi,
>>>>
>>>> The reason I'm skeptical about anything which binds processes/ports to
>>>> scheduler is that it feels like a temporary solution and would much
>>>> rather
>>>> do a proper solution where the scheduler takes care of these things for
>>>> you.
>>>> But as I said, internally we need to talk this over when it is not in the
>>>> middle of summer vacation.
>>>>
>>>> I did some benchmarking using ab and found basically the same figures as
>>>> you. The below is with keep-alive and the values are requests per second:
>>>>
>>>>                      not-bound        bound
>>>>
>>>> R15B01                 44k        37k
>>>>
>>>> master                  44k        35k
>>>>
>>>> master+mp          48k        49k
>>>>
>>>> master+mp+pb    49k        55k
>>>>
>>>> [mp]: multi-poll patch
>>>> [pb]: port bind patch
>>>> [bound]: Used {scheduler,I} to spread load
>>>>
>>>> Unfortunately I also found that when doing the non-keep alive benchmark
>>>> the
>>>> performance is seriously degraded.
>>>>
>>>> R15B01 not-bound                  8255
>>>> master+mp+pb not-bound    7668
>>>> master+mp+pb bound           5765
>>>>
>>>> I did some gprof runs but could not find anything obvious that is going
>>>> wrong.
>>>>
>>>> Lukas
>>>>
>>>>
>>>> On 11/07/12 04:21, Wei Cao wrote:
>>>>
>>>> I added a macro to conditional compile the patch because I think it
>>>> can be more selectable, I can remove the macro, fix the compilation
>>>> error and test on mingw platform in later version.
>>>>
>>>> how about provide another BIF named port_flag (like process_flag) to
>>>> let user bind port to a given scheduler?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>