TCP stack throughput

[Ulf Wiger]

Den 2005-07-19 19:39:12 skrev Javier París Fernández <paris@REDACTED>:

> I observed several funny things. The time spent processing the
> connections seems to go up exponentially, but it is not limited
> by cpu, which was always under 50%. I don't know what is throttling it.
>
> When I trid with 5000 connections, the logs started to show  
> retransmissions,
> so I think it is probably related to some buffering limit with
> the socket. This obviously skyrocketed the total processing time.

Have you tried playing with setsockopt()?

This seems to be the way to set socket send and receive buffer
sizes on most platforms. Parameters are SO_SNDBUF and SO_RCVBUF

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/setsockopt_2.asp
http://docs.hp.com/en/36960-90058/ch04s05.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0102.0/0038.html


On Linux, the buffer sizes "can be set globally with the
net.ipv4.tcp_wmem and net.ipv4.tcp_rmem sysctl variables"
(http://unixhelp.ed.ac.uk/CGI/man-cgi?tcp+7)


The defaults seem to vary. On HP AIX, the default is
4096. On Windows CE, it's 32768
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/setsockopt_2.asp)


/Uffe

>
> The total processing time for the tests was:
> 500 in 0.6 seconds
> 1000 in 1.7 seconds
> 2000 in 6.2 seconds
> 5000 was still closing connections after 3 minutes
>
> There are tcpdump logs in
> http://www.lfcia.org/~paris/dump500.gz
> http://www.lfcia.org/~paris/dump1000.gz
> http://www.lfcia.org/~paris/dump2000.gz
> http://www.lfcia.org/~paris/dump5000.gz
>
> I think it would be interesting to find what is the limiting the stack
> here, and why there are dropped packets, as I think it should be
> possible to get better performance. However, I had not made a test
> similar to this, so connection opening and closing is one of the things
> that are not optimized in the stack.
>
> Regards,
> Javier.
>