[erlang-questions] gen_tcp:connect taking 3 seconds under load

[Adam Kocoloski]

On Apr 15, 2010, at 8:56 PM, Jacob Vorreuter wrote:

> Hi,
> 
> I'm load testing an Erlang proxy server that accepts incoming client connections and proxies data to backend server sockets.  Normally, the call to gen_tcp:connect to open the backend socket takes under 3 milliseconds.  However, under heavy load this call sometimes takes 3 seconds.  The strange thing is that it rarely takes any amount of time in the middle of that range.  It's either 3 milliseconds or 3000.  This is the call I'm making:
> 
> gen_tcp:connect(Ip, Port, [binary, {packet, 0}]).
> 
> Has anyone seen this behavior before?  I'm load testing with ab, making anywhere from 64 to 1000 concurrent requests.
> 
> Thanks,
> 
> Jake

Hi Jake, 3 seconds is a magic number in TCP -- it's the TCP retransmission timeout.  You might double check for an abnormally high number of lost packets, but the last time I had this happen I was able to solve the problem by increasing the listen backlog on the backend.  If you're using gen_tcp:listen on the backend you can supply the {backlog, integer()} option, but be sure to check that your backend's SOMAXCONN setting will allow the value you supply.  Best,

Adam