Scheduler threads high CPU utilization on Erlang 11.0.3

Gopi V gopi1828@REDACTED
Fri Oct 9 18:56:58 CEST 2020


Note: Reposting this as I haven't got any response.

I've upgraded from rabbitMQ version: 3.4.4 on Erlang R16B03 (erts-5.10.4)
to rabbitMQ 3.8.5 on Erlang 11.0.3 (OTP 23).
When I ran rabbitmq-perf-test on 3.8.5 it was able to handle 1000's of
messages per second with 500 producers and 500 consumers. (
https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle). I did see
CPU utilization around 500%-600% on 8 CPU system whereas it stayed below 1
CPU utilization for 100-200 messages/sec which is the case of 3.4.4
production cluster.
But in production 3.8.5 is taking 50% more CPU when compared to 3.4.4.

Below is the output of rabbitmq-diagnostics runtime_thread_stats which is
taking more time on other (I assume its busy wait). The default value of
scheduler busy wait (sbwt) is medium on both the version of erlang. When I
changed this to None the CPU utilization went down from 4-5 CPU's to 1 CPU,
when I tried very_short then CPU utilization is same as 3.4.4 which has
sbwt set to medium (default from erlang docs). Is this a know issue?

Average thread real-time : 5000228 us
Accumulated system run-time : 12160527 us
Average scheduler run-time : 1511950 us

    Thread      aux check_io emulator       gc    other     port    sleep
Stats per thread:
scheduler( 1) 0.47% 0.52% 7.89% 0.54% 52.64% 1.27% 36.68%
scheduler( 2) 0.53% 0.49% 7.15% 0.52% 51.70% 1.11% 38.51%
scheduler( 3) 0.50% 0.50% 7.46% 0.51% 52.58% 1.17% 37.28%
scheduler( 4) 0.40% 0.35% 5.33% 0.36% 46.75% 0.79% 46.01%
(ignoring scheduler 5 to 8)


Setting below value will for sure reduce CPU utilization.

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+sbwt very_short +sbwtdcpu very_short
+sbwtdio very_short"

or

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+sbwt none +sbwtdcpu none +sbwtdio
none"

But I want to know
erts-5.10.4 has only *sbwt flag with *default value as medium (
http://erlang.org/documentation/doc-5.10.4/erts-5.10.4/doc/html/erl.html).
Erlang 11.0.3 has *sbwt flag with *default value as medium (
http://erlang.org/documentation/doc-11.0-rc3/erts-11.0/doc/html/erl.html#+sbt)
but also introduced *sbwtdcpu and **sbwtdio* with default value as short.
>From settings nothing has changed with scheduler busy wait threshold and I
do see twice CPU utilization with 11.0.3.
1. Does anyone know there are significant amount of change in scheduling
that would cause this?
2. If yes, then apart from consuming CPU when traffic is sparse does this
have any negative impact when traffic is continuous and number of messages
doubles? Technically,  in this scenario scheduler threads won't be in busy
wait state.
3. I tried to replicate my production scianrio with 500 messages/secs for
10 secs and keeping it idle for 3 secs but rabbitMQ is only utilizing 1 CPU
and less 10% on of busy wait. I want to replicate a scenario where for a
workload 3.8.5 has 50% on busy wait consuming x CPU's while 3.4.4 consumes
x/2 CPU's. Then go back to 3.8.5 and set *sbwt to very_short *and run the
test again to see if there was any significant impact on throughput.

./runjava com.rabbitmq.perf.PerfTest --queue-pattern 'perf-test-%d'
--queue-pattern-from 1 --queue-pattern-to 6 --producers 500 --consumers 500
--json-body --size 16000 --body-content-type application/json
--variable-rate 1:10 --variable-rate 0:3


-- 
Thanks,
Gopi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20201009/dae6986b/attachment.htm>


More information about the erlang-questions mailing list