massive tcp servers

[Shawn Pearce]

Dustin Sallings <dustin@REDACTED> wrote:
> 
> 	Does anyone have any experience with massively large tcp servers?  
> 	I'm doing something like a chat server where there will be many, many 
> connections simultaneously along with some mechanism for addressing 
> those connections.  Does anyone have any idea what might be required to 
> have something on the order of 10,000,000 concurrent connections in a 
> cluster.  Obviously I want as small of a cluster as possible for 
> hardware costs.  I also get the impression that large clusters may not 
> be that easy to scale.

http://www.sics.se/~joe/apachevsyaws.html

Erlang will easily take 80,000 connections on some OSes and still
keep a pretty good throughput.

10 million connections may take quite a few machines; I'd expect
you would want to be hitting around 100k-200k TCP connections per
physical computer, with about 10k-80k TCP connections per Erlang node.
100k/computer = 100 computers.

But what do I know. :)

Its really a function of how much work you need each to do, Erlang's
IO system on some platforms is pretty capable of going to large
numbers of file descriptors.  On others, I think its lucky to get
1024 without making the C libraries choke (FD_SET size limits).

The real limiting factor might just be the CPU load of your application,
requiring you to use even more systems than you would like.  Because
if you do much more than just act as a data pump, most of your nodes
will be overwhelmed with 80k connections.

I assume you have at least researched IRC?  It might not entirely fit
with your problem domain, but it does sort of fit into the "building a
very big cluster to support a very large number of connections from a
very large number of clients" arena.  At the very least it offers some
lessons learned, and some good do's and don'ts.

-- 
Shawn.