[erlang-questions] Erlang under high TCP/IP load
Mon Dec 11 14:51:51 CET 2006
Anders Trondheim wrote:
> I've been experimenting with ejabberd and Tsung. The setup consisted of two
> Erlang nodes on old PCs (Low spec) being bombarded with requests from a
> stationary machine and a laptop (Both of a good spec). By finetuning the
> parameters passed to the emulator (IO threads, Kernel Poll, etc), I managed
> to get a throughput increase of about 30%.
> At this point, the CPU usage was about 70% when the errors started occuring,
> and slowing down the connection rate resulted in a higher number of
> simultaneously users. The slave node on another machine helped bring up the
> load, but still hitting a limit before the machine hit 100% CPU usage. It
> looks like the system resources TCP/IP stack is the bottleneck, as the
> system resources were maxed out.
> I am wondering about the following:
> * Is there an easy way to see how much of the network overhead is caused by
> the distributed Mnesia and how much is caused by the test clients?
The most of this mail is about FreeBSD :)
FreeBSD have trafshow in ports - which is your friend (you can monitor
single connection to slave, which is 1 tcp connection, remaining traffic
is you client's traffic). But it slows dows a system a little bit -- so
the most wise is to run separate network analyzer or run network
analyzer on slave machine and systat/netstat -w 1 on master.
Substracting these numbers you can get clients traffic.
Also systat in IP stack and TCP display mode can greatly help with
> * Is there a preferred OS on which I can run clustered Erlang Nodes under
> heavy TCP/IP traffic, with thousands of socket connections open? If so, what
> parameters should be finetuned? Free BSD with a kernel poll flag has been
> suggested, but the person had no first hand experience.
Afaik, there was a problem with FreeBSD/ERTS -- something was broken
with kpoll ? Had anyone fixed that? Not sure.
In FreeBSD (and I'm sure, in any other *nix OS), you should decrease
network buffer sizes to minimal allowed (I think it's 4Kb - page size),
if you are serving a lot of slow clients, with little traffic from each
one. Also it may be necessary to tune max size of network buffers
allocated in kernel (FreeBSDish -- kern.ipc.nmbclusters -- Set the
number of mbuf clusters to be allocated).
I can suggest also Solaris as target OS, because it have a lot of
network stack tuning variables. But in that case you should have
experienced Solaris admin on hands ;) because it's too easy to degrade
performance instead of increasing it.
> I emailed the ejabberd mailing list, but got no helpful response. As this is
> not an IM specific domain, hopefully someone on the list can give me some
> Anders Trondheim
Network security consulting
t +37493 419763 (mob)
More information about the erlang-questions