[erlang-questions] Heavy duty UDP server performance

[Ameretat Reith]

On Thu, 11 Feb 2016 10:22:55 +0100
Lukas Larsson <lukas@REDACTED> wrote:

> Would you mind sending me the "perf script" output from your
> profiling runs so that I can have a closer look?

[1] is for E3 case which used 50% of CPU to take 1Gbit/s traffic.

> 
> For the record, CPU utilization is a terrible way to measure load on a
> system, especially if you are running on much less than full throttle.
> Because of how expensive it is to go to sleep, all the threads in the
> Erlang VM will spin and consume CPU when it is out of work and if you
> are running at 25% CPU then you are running out of work very often
> which will cause more spinning which will increase CPU while not
> decreasing the amount of work the system can do.

While I was monitoring CPU on above test case,  All cores had even and
stable load, so I think scheduler turning off and on was not an issue
here.  I didn't included sasl in this testing application but for my
real application I hardly can see scheduler utilization jumps when CPU
core usage is stable around 50%.  I agree that's not really useful for
performance analysis since much time would spent for polling in that
case.

I made other samples in my local machine, server and client running on
one machine, server opens N ports for client and client responds to each
packet came from client, then client sends another packet when he get
response for previous packet.  I told VM to open 2 schedulers, (+S2 in
vm.args).  CPU is dual core with hyperthreading on. 

I open just open port, CPU usage is about 20%, cores utilization is not
even and each one jump between 0 to 60.  System load is stick to 1.4.
Receive rate is about 290Mbit/s.  perf script [2]. 

Opening two ports, CPU usage become around 50, still core usage jumps
but this time more just 0 or 100.  I don't know why perf saw just one
scheduler.  System load became stable at 1.8.  Receive rate is about
690Mbit/s, much better than double of previous run.  perf script [3].

By 10 ports, CPU usage is 100%, system load keep increasing and after 5
minutes seems won't exceed 3.9.  Top CPU consumer is process_main by
11%.  Receive rate reach 750Mbit/s. perf script [4].

To see how system react by much more load, I opened 1000 ports.  Run
queue sometimes reach 600 but on average is about 16 in server.  Receive
rate drops to 520Mbit/s. perf script [5]

Thanks.


1:http://file.reith.ir/erludp/perf-script-e3-1gbit-50.out.bz2
2:http://file.reith.ir/erludp/perf-script-local-i3-4170-293mbits-1x.out.bz2
3:http://file.reith.ir/erludp/perf-script-local-i3-4170-690mbits-2x.out.bz2
4:http://file.reith.ir/erludp/perf-script-local-i3-4170-750mbits-10x.out.bz2
5:http://file.reith.ir/erludp/perf-script-local-i3-4170-520mbits-1kx.out.bz2