[erlang-patches] Pollset per scheduler and bind port to scheduler

[Lukas Larsson]

Hi!

You should disable hipe before trying to compile the debug build. Also 
I'm unsure if just adding -DDEBUG to CFLAGS is enough. To build a debug 
emulator the way we do it is:

(cd $ERL_TOP && ./otp_build autoconf && ./otp_build configure 
--disable-hipe && ./otp_build boot)
(cd $ERL_TOP/erts/emulator && make FLAVOR=smp debug)

and then start it using

$ERL_TOP/bin/cerl -debug

Lukas

On 18/07/12 08:23, Wei Cao wrote:
> 2012/7/18 Wei Cao <cyg.cao@REDACTED>:
>> 2012/7/17 Lukas Larsson <lukas@REDACTED>:
>>> Hi!
>>>
>>> After fixing the patch to compile on windows[1] I put it in our daily builds
>> Glad to hear it can compile on windows now, thanks a lot~ I tried to set up
>> mingw etc on a windows virtual machine yesterday, but it runned so slow, and
>> met a lot of mistakes during configuration.
>>
>>> and a couple of issues came up. I did not include 'move erts_deliver_time
>>> out of erl_check_io' commit as it deadlocks the non-smp emulator.
>> Yeah, I can confirm this problem, there is only one OS thread
>> innon-smp emulator,
>> if check_io starts waiting, other threads will have no opportunity to
>> run, including the
>> thread I added to call erts_deliver_time() periodically, so check_io
>> will have no timer
>> to wake it up, and would wait endlessly.
>>
>> I'll consider to replace a better solution.
>>
>>> When running a debug test build on Linux we got the following assertion:
>>> Assertion failed: 0 <= (ix) && (ix) < erts_no_pollsets in
>>> sys/common/erl_check_io.c, line 1909
>> hm... this is a very silly mistake, the variable erts_no_pollsets is
>> set after the assertion,
>> after fixing it(post to pollset_per_scheduler branch), I met another
>> assertion error,
>>
>> erlc -W  +debug_info +inline -o../ebin hipe_icode2rtl.erl
>> Assertion failed: VALID_INSTR(* (Eterm *)(c_p->i)) in beam/beam_emu.c, line 1241
>>
> I tried the latest master branch on erlang/otp(without my patch), configure with
> ./configure CFLAGS="-DDEBUG -g -O3 -fomit-frame-pointer"
> there is the same assertion failure:
> Assertion failed: VALID_INSTR(* (Eterm *)(c_p->i)) in beam/beam_emu.c, line 1241
>
> However, if instead I configure with
> ./configure CFLAGS="-DDEBUG", above assertion failures disappear,
> instead reporting warnings like this:
>
> bit length overflow
> code 12 bits 7->6
>
> It seems CFLAGS influence these assertions.
>
>>> On OS X Lion we got the following when compiling gs:
>>> erl -pa ../ebin -s gs_make -s erlang halt -noshell
>>> ../include/internal/ethr_mutex.h:655: Fatal error in ethr_mutex_lock():
>>> Invalid argument (22)
>>> make[3]: *** [gstk_generic.hrl] Abort trap: 6
>>> make[2]: *** [opt] Error 2
>>> make[1]: *** [opt] Error 2
>>> make: *** [libs] Error 2
>> I didn't have a mac machine to test it.
>>
>>> I'll remove the branch and see if the same problems appear again tomorrow
>>> (unfortunately we do not have enough machines to let your branch run alone,
>>> so it might be some other branch causing this). Let me know if you need any
>>> help tracking down these issues.
>>>
>>> Lukas
>>>
>>> [1]: https://github.com/garazdawi/otp/tree/wc/pollset_per_scheduler
>>>
>>>
>>> On 11/07/12 17:34, Wei Cao wrote:
>>>
>>> In non keep-alive cases, all new connections 're accepted by the
>>> Erlang port which listens on the TCP port, and how frequently/fast the
>>> port be scheduled to run limits the QPS. (requests per second), so
>>> this port can be regarded as bottleneck of non keep-alive
>>> applications.
>>>
>>> So I guess performance degradation observed is caused by the listener
>>> port not be scheduled frequent or fast enough, I'll look into this
>>> problem tomorrow, now is at night in China, :-)
>>>
>>> BTW, I found this patch should be compiled like this today,
>>> ./configure CFLAGS="-DERTS_POLLSET_PER_SCHEDULER -g -O3
>>> -fomit-frame-pointer"
>>> otherwise compiler optimization is disabled.
>>>
>>> Regarding binding processes/ports to scheduler, I admit it's really a
>>> temporary solution to bind port to the same scheduler as its owner
>>> process like the pb patch did, and I suggest it's better to add a
>>> additional BIF like erlang:process_flag, to allow user explicitly bind
>>> port to a given scheduler, if it benefits.
>>>
>>>
>>> 2012/7/11 Lukas Larsson <lukas@REDACTED>:
>>>
>>> Hi,
>>>
>>> The reason I'm skeptical about anything which binds processes/ports to
>>> scheduler is that it feels like a temporary solution and would much rather
>>> do a proper solution where the scheduler takes care of these things for you.
>>> But as I said, internally we need to talk this over when it is not in the
>>> middle of summer vacation.
>>>
>>> I did some benchmarking using ab and found basically the same figures as
>>> you. The below is with keep-alive and the values are requests per second:
>>>
>>>                     not-bound        bound
>>>
>>> R15B01                 44k        37k
>>>
>>> master                  44k        35k
>>>
>>> master+mp          48k        49k
>>>
>>> master+mp+pb    49k        55k
>>>
>>> [mp]: multi-poll patch
>>> [pb]: port bind patch
>>> [bound]: Used {scheduler,I} to spread load
>>>
>>> Unfortunately I also found that when doing the non-keep alive benchmark the
>>> performance is seriously degraded.
>>>
>>> R15B01 not-bound                  8255
>>> master+mp+pb not-bound    7668
>>> master+mp+pb bound           5765
>>>
>>> I did some gprof runs but could not find anything obvious that is going
>>> wrong.
>>>
>>> Lukas
>>>
>>>
>>> On 11/07/12 04:21, Wei Cao wrote:
>>>
>>> I added a macro to conditional compile the patch because I think it
>>> can be more selectable, I can remove the macro, fix the compilation
>>> error and test on mingw platform in later version.
>>>
>>> how about provide another BIF named port_flag (like process_flag) to
>>> let user bind port to a given scheduler?
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>>
>> Best,
>>
>> Wei Cao
>
>