massive tcp servers

[Luke Gorrie]

Dustin Sallings <dustin@REDACTED> writes:

> 	Does anyone have any experience with massively large tcp
> servers?  I'm doing something like a chat server where there will be
> many, many connections simultaneously along with some mechanism for
> addressing those connections.  Does anyone have any idea what might be
> required to have something on the order of 10,000,000 concurrent
> connections in a cluster.  Obviously I want as small of a cluster as
> possible for hardware costs.  I also get the impression that large
> clusters may not be that easy to scale.

Oh, oh, oh, that sounds like fun!

I'm assuming that most of these users will be idle most of the time,
and each will use a small amount of overall bandwidth.

Entering fantasy land here..

My first thought is that the OS will have trouble with that many
connections. I'm not certain of the per-connection overhead, but
looking at the 'tcp_opt' struct in include/linux/tcp.h at least it
looks significant.

If you wrote your own TCP (or simpler custom protocol) in userspace
then maybe you could pull it off with just one box. You could keep the
TCP control structures in a database on disk and map the active ones
into a large-but-bounded-size pool in memory.

I have only limited experience with hacking TCP/IP, but my impression
is that writing your own TCP is probably no harder than e.g. writing a
good HTTP/1.1 implementation. I recommend the first two volumes of
Douglas Comer's networking series, in turn recommended to me by Tobbe
from when he wrote a TCP in Erlang.

On Linux I think the most practical option would be to use a 'tun'
interface (linux/Documentation/networking/tuntap.txt), which allows
you to write a user-space network interface that operates at IP-level
(rather than ethernet-level). That way you can use Linux's IPv4
implementation and only worry about TCP yourself.

-Luke