[erlang-questions] Erlang under high TCP/IP load

[Gaspar Chilingarov]

Anders Trondheim wrote:
> I've been experimenting with ejabberd and Tsung. The setup consisted of two 
> Erlang nodes on old PCs (Low spec) being bombarded with requests from a 
> stationary machine and a laptop (Both of a good spec). By finetuning the 
> parameters passed to the emulator (IO threads, Kernel Poll, etc), I managed 
> to get a throughput increase of about 30%.
> 
> At this point, the CPU usage was about 70% when the errors started occuring, 
> and slowing down the connection rate resulted in a higher number of 
> simultaneously users.  The slave node on another machine helped bring up the 
> load, but still hitting a limit before the machine hit 100% CPU usage. It 
> looks like the system resources TCP/IP stack is the bottleneck, as the 
> system resources were maxed out.
> 
> I am wondering about the following:
> 
> * Is there an easy way to see how much of the network overhead is caused by 
> the distributed Mnesia and how much is caused by the test clients?

The most of this mail is about FreeBSD :)

FreeBSD have trafshow in ports - which is your friend (you can monitor 
single connection to slave, which is 1 tcp connection, remaining traffic 
is you client's traffic). But it slows dows a system a little bit -- so 
the most wise is to run separate network analyzer or run network 
analyzer on slave machine and systat/netstat -w 1 on master.
Substracting these numbers you can get clients traffic.
Also systat in IP stack and TCP display mode can greatly help with 
system tuning.

> 
> * Is there a preferred OS on which I can run clustered Erlang Nodes under 
> heavy TCP/IP traffic, with thousands of socket connections open? If so, what 
> parameters should be finetuned? Free BSD with a kernel poll flag has been 
> suggested, but the person had no first hand experience.
> 

Afaik, there was a problem with FreeBSD/ERTS -- something was broken 
with kpoll ? Had anyone fixed that? Not sure.
In FreeBSD (and I'm sure, in any other *nix OS), you should decrease 
network buffer sizes to minimal allowed (I think it's 4Kb - page size), 
if you are serving a lot of slow clients, with little traffic from each 
one.  Also it may be necessary to tune max size of network buffers 
allocated in kernel (FreeBSDish -- kern.ipc.nmbclusters -- Set the 
number of mbuf clusters to be allocated).

I can suggest also Solaris as target OS, because it have a lot of 
network stack tuning variables. But in that case you should have 
experienced Solaris admin on hands ;) because it's too easy to degrade 
performance instead of increasing it.



> I emailed the ejabberd mailing list, but got no helpful response. As this is 
> not an IM specific domain, hopefully someone on the list can give me some 
> pointers.
> 
> Hilsen,
> Anders Trondheim
> 


-- 
Gaspar Chilingarov

System Administrator,
Network security consulting

t +37493 419763 (mob)
i 63174784
e nm@REDACTED