[erlang-questions] listen() backlog limited to 16 bits?
Tue May 5 22:39:01 CEST 2009
Some comments inline.
Matthias Lang wrote:
> On Monday, April 27, Matthew Dempsky wrote:
>> My only point is that (AFAIK) the listen backlog doesn't directly
>> affect the number of requests per second or the number of concurrent
>> connections you can handle. It's just the number of TCP connections
>> that have completed the SYN/ACK handshake and not yet been accepted by
>> user space, and I just wouldn't expect this number to get into the
>> hundreds of thousands unless the system is heavily overloaded, and
>> then it probably can't do anything useful with such a huge backlog
> I had similar thoughts. I can imagine a short backlog limit, e.g. 5,
> limiting throughput, but I can only think of contrived situations
> where there'd be a difference between millions and thousands.
My experiments with this is that there is a very close relation between
the backlog and how many connections / second you can handle. When we
tested it though we had very bursty clients, essentially a C program
that would loop out X connections and *then* start to use them. The
interesting problem here is something that you have described later but
that I don't fully understand yet. As soon as our backlog became full
(you can read some statistics with netstat --statistics, try greping for
"times the listen queue of a socket overflowed"), any new connection
attempt seem to take a very long time. Maybe this is because of some syn
flood protection in the Linux TCP stack. In this case is it possible to
turn this of, at least during load testing, when all connections will
come from very few hosts and look very much like dos attacks.
More on overflowing listen queues further down :)
> The OS backlog limit has increased in linux over the
> years. Originally, it was five connections (BSDism?). Later
> (2.0/2.2/2.4?), it increased to 128. In current Linux versions it's
> configurable without an apparent upper limit; the default is 128.
> Someone thought it was worth the effort to remove the limit. I googled
> a bit and couldn't find the reasoning behind that. Maybe it was a
> "remove all arbitrary limits" mission.
> * I've tried writing higher values to somaxconn and reading them
> back again. It gets a bit weird for some values:
> contorpis:/proc/sys/net/core# echo "2147483648" > somaxconn
> contorpis:/proc/sys/net/core# cat somaxconn
> I also tried a C program with various backlog arguments. It feels
> like something strange is going on, e.g. with a call to listen(s,
> 1) and then sleep() forever, I can still connect several sockets,
> though it takes about a second per connection after the first
> few. I wasn't sufficiently interested to figure out why, but if
> someone knows, it'd be interesting to hear.
This can be because of syncookies. Syncookies is essentially a way of
avoiding dropping connection attempts even if syn floods are send to a
server. I'm guessing that since your backlog is negative, you can't
accept connections, but by using syn cookies, the client believes that
the connection has been accepted.
As far as I've understood it (from googling), a TCP stack with
syncookies enabled will first fill up its backlog. When that has
happened it will start sending syn cookies back to clients that are
trying to connect. A syn cookie is essentially a magic hash of the
source IP and port of the connection attempt, which is send back as the
initial sequence number on the server side. For the client it will look
like the application on the other side accepted the connection and the
it should send an ACK to the server. If the ack is for the right
sequence, and that matches the has for the source IP and port of this
ACK, the application can then use the connection, if it is running accept.
On linux the following configuration is relevant:
net.ipv4.tcp_syncookies (enable / disable syncookies)
net.ipv4.tcp_max_syn_backlog (the number of pending connections allowed
with syn cookies)
Don't know if this helps to shed some light?
Phone: +44 (0)798 45 44 773
Mobile: +44 (0)207 65 50 337
More information about the erlang-questions