[erlang-questions] Yaws and httpc
Sat May 26 18:13:13 CEST 2012
I have a web client written using "httpc" module, and I have Yaws web
server with all default configurations except for "listen_backlog", which
is set to 10000 (ten thousand).
I am not sure how to program error-recovery for a test-client application
that uses "httpc" module when httpc:request/2 encounters an error as
>>>> Error in process <0.17159.0> with exit value:
I presumed I would close the the dynamically created profile [*1] using
inets:stop/2, and restart it with the same profile-name using
inets:start/2, and submit the http request again using httpc:request/2.
But, then, it encounters this error:
>>>> Error in process <0.11323.0> with exit value:
Basically, the test-client application spawns 10000 (ten thousand)
processes [*2], each creating its own profile with max_sessions = 1 and
max_keep_alive_length = 1 [*3], and submitting the transactions
synchronously in a loop with an internal of 20 seconds between each
submission. The first submission for all processes go through
successfully; but, from second submission onwards, some of the client
processes start receiving "socket_closed_remotely" errors as shown above.
Please advise on how to recover from this error and resume the
transactions on the client side [*4]
I tried altering Yaws configuration parameter keepalive_timeout from 30000
to 120000. I also tried keeping an internal (60 seconds) between stopping
and restarting the client profile upon encountering the socket failures.
But that didn't alter the error behaviours seen at the client side [*5]
My assumption is that Yaws should not close the socket because the client
takes care of not keeping the connection idle for more than 20 seconds.
[*1] The test-application needs to simulate many "web browser" clients, so
I suppose I keep a separate profile for each client process.
[*2] Both Erlang web client and Yaws (1.92) web server are running on
R15B01 (erts-5.9.1) and on 32-bit Ubuntu Server 12, and on separate
machines. ERL_MAX_PORTS, ERL_MAX_ETS_TABLES and "ulimit -n" are set to 64K
on both client & server machines.
[*3] My assumption is that "httpc" inets service should not open more than
one connection to web server since test-client is submitting the
transactions synchronously. But, I am seeing that the number of
connections to the web server grow more than 16000 even when there are
only 10000 "httpc" services running in the inets container.
[*4] Yaws logs show no abnormal activities when "httpc" services on the
client receive sockets failures, except for information on reaching the
maximum number of files in cache.
[*5] I have raised the number of processes to 500000 using +P only on the
client side ERTS.
More information about the erlang-questions