[erlang-questions] Maximize Client TCP Sockets

[Danil Zagoskin]

Hi!

What is your current problem with lots of connections?

Just in case, here are problems I've hit:
  - (emfile) max ports beam option: erl +Q 2000000 (check with
'erlang:system_info(port_limit)')
  - (emfile) ulimit (OS/shell feature): check 'ulimit -n'. Make it large
enough (use google). Check 'max_fds' field in 'erlang:system_info(check_io)'
  - (eaddrnotavail) local TCP ports exausted: on Linux, check 'sysctl
net.ipv4.ip_local_port_range'. Increase it. Use '{reuseaddr, true}' option
and manually distribute local ports (one local port is OK to use with
different remote {Addr, Port}s).

More info on a problem with local TCP ports exausted:
  - Try TCP_TW_REUSE option
http://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both
  - Use multiple addresses on your network interface (it's easy with IPv6).
Use round-robin or random local address for every new connection.


And if you use HTTP(S) to connect to a limited number of servers, it's
better to use keep-alive connections. With them you don't spend time on TCP
handshake and use much fewer local ports.

On Mon, Jan 18, 2016 at 1:33 AM, scdelta1249 <scdelta1249@REDACTED> wrote:

> I have an application where I need to make 10s of millions of short-lived
> plain TCP (gen_tcp) or HTTP/HTTPS (httpc/gun/hackney) requests to different
> servers. The parameters for the requests are stored in a database. My plan
> is for a controlling gen_server on each node in the cluster to:
>
> * grab a group of requests from the database
> * spawn a process for each request
> * the spawned process connects, makes the request, processes the response
> and shuts down the connection
>
> What is the best strategy to maximize the number of concurrent clients? My
> current plan is to maintain a list of pending requests in the controlling
> process (gen_server). As processes complete, they remove themselves from
> the list. If they error out on initialization I'd implement some sort of
> back-off timer, pushing the unexecuted requests back into the database if
> they couldn't execute after a predefined time limit.
>
> Is there a better way to maximize the number of TCP client connections per
> node in Erlang?
>
> Thanks.
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>


-- 
Danil Zagoskin | z@REDACTED
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160118/1d65132d/attachment.htm>