[erlang-questions] Throughput on Yaws 1.68 / OTP R11B-3 / Suse Linux 10.2?

[Francesco Cesarini]

We've been running some stress tests on a system receiving http posts 
from a load tool. No matter how hard we push the system, and no matter 
how many instances of the load tool we install, we can not increase the 
throughput.

We are hitting each web server node with 8,000 simultaneous inets http 
client connections (ibrowse, at those levels generates huge message 
queues and runs @ 100% cpu), sending messages as quickly as they can. 
The load machines are running at 10 - 15% system CPU. Once the requests 
have passed through the inets driver and the yaws layer, we are unable 
to handle more than 1000 sustained hits per second. The hardware is 
running at about 50% System CPU.  K-poll is enabled, as are 255 io 
threads. Hitting the system in the API used by Yaws, we have managed to 
achieve a sustained throughput in excess of 3,000 requests per second.

We are in the process of replacing Yaws with a light weight web server, 
but our gut feeling is that the bottle neck is in the inets driver. 
Benchmarks using the pico server a few years back (on much more modest 
hardware) resulted in excess of 3,000 requests per second. While we are 
doing it, it would be useful to hear if others have experienced similar 
bottlenecks in similar architectures?

Thanks,
Francesco
--
http://www.erlang-consulting.com