[erlang-questions] listen() backlog limited to 16 bits?

[Oscar Hellström]

Hi,

Some comments inline.

Matthias Lang wrote:
> On Monday, April 27, Matthew Dempsky wrote:
> 
>> My only point is that (AFAIK) the listen backlog doesn't directly
>> affect the number of requests per second or the number of concurrent
>> connections you can handle.  It's just the number of TCP connections
>> that have completed the SYN/ACK handshake and not yet been accepted by
>> user space, and I just wouldn't expect this number to get into the
>> hundreds of thousands unless the system is heavily overloaded, and
>> then it probably can't do anything useful with such a huge backlog
>> anyway.
> 
> I had similar thoughts. I can imagine a short backlog limit, e.g. 5,
> limiting throughput, but I can only think of contrived situations
> where there'd be a difference between millions and thousands.

My experiments with this is that there is a very close relation between 
the backlog and how many connections / second you can handle. When we 
tested it though we had very bursty clients, essentially a C program 
that would loop out X connections and *then* start to use them. The 
interesting problem here is something that you have described later but 
that I don't fully understand yet. As soon as our backlog became full 
(you can read some statistics with netstat --statistics, try greping for 
"times the listen queue of a socket overflowed"), any new connection 
attempt seem to take a very long time. Maybe this is because of some syn 
flood protection in the Linux TCP stack. In this case is it possible to 
turn this of, at least during load testing, when all connections will 
come from very few hosts and look very much like dos attacks.

More on overflowing listen queues further down :)

> The OS backlog limit has increased in linux over the
> years. Originally, it was five connections (BSDism?). Later
> (2.0/2.2/2.4?), it increased to 128. In current Linux versions it's
> configurable without an apparent upper limit; the default is 128.
> 
> Someone thought it was worth the effort to remove the limit. I googled
> a bit and couldn't find the reasoning behind that.  Maybe it was a
> "remove all arbitrary limits" mission.
> 
> Matt
> 
>  * I've tried writing higher values to somaxconn and reading them
>    back again. It gets a bit weird for some values:
> 
>     contorpis:/proc/sys/net/core# echo "2147483648" > somaxconn 
>     contorpis:/proc/sys/net/core# cat somaxconn 
>     -18446744071562067968
> 
>    I also tried a C program with various backlog arguments.  It feels
>    like something strange is going on, e.g. with a call to listen(s,
>    1) and then sleep() forever, I can still connect several sockets,
>    though it takes about a second per connection after the first
>    few. I wasn't sufficiently interested to figure out why, but if
>    someone knows, it'd be interesting to hear.

This can be because of syncookies. Syncookies is essentially a way of 
avoiding dropping connection attempts even if syn floods are send to a 
server. I'm guessing that since your backlog is negative, you can't 
accept connections, but by using syn cookies, the client believes that 
the connection has been accepted.

As far as I've understood it (from googling), a TCP stack with 
syncookies enabled will first fill up its backlog. When that has 
happened it will start sending syn cookies back to clients that are 
trying to connect. A syn cookie is essentially a magic hash of the 
source IP and port of the connection attempt, which is send back as the 
initial sequence number on the server side. For the client it will look 
like the application on the other side accepted the connection and the 
it should send an ACK to the server. If the ack is for the right 
sequence, and that matches the has for the source IP and port of this 
ACK, the application can then use the connection, if it is running accept.

On linux the following configuration is relevant:
net.ipv4.tcp_syncookies (enable / disable syncookies)
net.ipv4.tcp_max_syn_backlog (the number of pending connections allowed 
with syn cookies)

Don't know if this helps to shed some light?

Cheers
-- 
Oscar Hellström, oscar@REDACTED
Phone:  +44 (0)798 45 44 773
Mobile: +44 (0)207 65 50 337
Web:    http://www.erlang-consulting.com