[erlang-questions] CPU load of TCP server

[Rapsey]

Every time after gets executed a timer gets created (I presume). With 10k
processes it probably makes a noticeable CPU impact.


Sergej

2009/10/13 Andrey Tsirulev <andrey@REDACTED>

> Hello all,
>
> I'm exploring the possibility of using Erlang for my TCP service
> application (actually the game server). I've prepared test server and client
> applications. The test server application accepts client connections and
> sends 2 small (<1 Kb) packets per second to each client (and receives
> answers).
>
> I've met the following problems:
> 1) Kernel polling doesn't give any benefit with R13B02-1.
> 2) CPU load is too high.
>
> All the details are below.
>
> Here's my test server's `uname -a`:
> Linux source 2.6.29-gentoo-r5 #1 SMP Tue Aug 18 01:15:17 MSD 2009 x86_64
> AMD Sempron(tm) Dual Core Processor 2200 AuthenticAMD GNU/Linux
> (I've made tests also with 2 other linux servers with different kernel
> versions and results were close).
>
> I've made server connection processes as simple and possible. I've tried up
> to 10000 concurrent connections.
>
> Test results didn't not show any visible difference between using multiple
> remote machines for client connections, one remote machine or localhost.
>
> I tried R13B02-1 and R12B-5 OTP versions.
>
> I found that memory usage grow is linear, as expected. But I came to the
> problem with CPU load.
>
> First of all, kernel polling didn't give any benefit for R13B02-1 (while
> erlang:system_info(kernel_poll) returned true and erl started with message
> [kernel-poll:true]). I've got about 55% of CPU usage with 4000 connections
> both with and without kernel polling enabled, while with R12B-5 I have about
> 26% of CPU usage with +Ktrue. I suspect a bug either in OTP or in gentoo
> ebuild (of course it's also quite possible that I'm doing something wrong or
> missed something in docs).
>
> The following is about R12B-5. I get about 6-7% of CPU load per every 1000
> connections (about 60% CPU load for 10000 connections). I'm not sure if I
> should consider this as a good result or a bad one. Most of the articles on
> the same subject say that CPU load is negligible in their tests and they are
> fighting for memory only, so I expected I won't be CPU-limited too, but
> evidently I am.
>
> `top` says that about 50% of CPU load is userspace, 25% software
> interrupts, 20% system and 5% hardware interrupts (that's by eye, not very
> strict).
>
> I found that CPU load depends not as much on connection count but on
> transmitted packet count (ok, that's obviously the number of system calls).
> Thus if I send 4 packets per second, not 2, I should decrease the number of
> connections twice to preserve the same CPU load.
>
> CPU load does not depend on packet size. 1 byte or 1Kbyte - no visible
> difference.
>
> CPU usage is slightly less with active socket option enabled than with
> blocking recvs.
>
> CPU usage on the single windows client machine with 4000 connections
> spawned is on the same level as with the linux server handling these 4000
> connections (while I expected linux to perform better).
>
> Switching Nagle on and off had no effect. I also tried to tune TCP stack
> with sysctl using advises found here and there but almost without any effect
> too.
>
> I've tried to trace with fprof and found that bottlenecks are 'send'
> operations (but I'm a relative novice to erlang so I'm not sure my usage of
> fprof was correct). Ok, that was expected too. I've read the 'why is
> gen_tcp:send slow?' thread but none of advises given there helped me.
>
> So the main question is: is the CPU usage of 7% per 1000 connections (or
> maybe better say 2000 packets per second) a good result? If no, what is the
> expected result? How can I improve my test application? Or maybe something
> in my story looks strange?
>
> I know that the possible optimization is decreasing the number of packets
> and keep it in mind.
>
> Here's the server connection process loop:
>
> loop(Socket) ->
>        receive
>                {tcp, Socket, _Packet} ->
>                        loop(Socket);
>                {tcp_closed, Socket} ->
>                        normal;
>                _ ->
>                        loop(Socket)
>        after 500 ->
>                        gen_tcp:send(Socket,[?PACKET]),
>                        loop(Socket)
>        end.
>
> Client loop has blocking recv and answers with send immediately.
>
> Thank you very much for your time. Sorry for too many words, I tried to
> provide all possible information. I will answer any question and appreciate
> any hint.
>
> Best regards,
> Andrey