<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.30.3">
</HEAD>
<BODY>
<TT><FONT COLOR="#3c3c3c">Hello,</FONT></TT><BR>
<BR>
First of all, I apologize if this question occurs twice. I have tried submitting it couple of hours ago via google groups, but it didn't show up, so I'm trying via e-mail now.<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">I am trying to develop https comet (long polling) server using mochiweb. I have already written one such http server which is in production</FONT></TT><TT> </TT><TT><FONT COLOR="#3c3c3c">f</FONT></TT><TT>or </TT><TT><FONT COLOR="#3c3c3c">c</FONT></TT><TT>ouple </TT><TT><FONT COLOR="#3c3c3c">o</FONT></TT><TT>f </TT><TT><FONT COLOR="#3c3c3c">m</FONT></TT><TT>onths</TT><TT><FONT COLOR="#3c3c3c"> and serves no more than 1000 concurrent users, sending broadcast message to all concurrent users at an average rate of 0.3 broadcast/sec</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">When I added ssl support, it essentially works when smaller number of users (about 100-200) are connected. With larger number, it stops responding correctly i.e. some connection attempts are refused, requests time out etc.</FONT></TT><TT> Only ssl/https part seems to be affected by this behavior.</TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">To better test this, I developed small test mochiweb app which works as follows:</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">1. Spawn and register singleton broadcast process</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">2. For each request, the erlang process sends its pid to the broadcast process. Then it waits for the message from the broadcast process</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">3. Every 5 seconds broadcast process notifies all registered https response processes from 2, then it clears its list of registered processes</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">4. When process from 2 receives notification, it responds and finishes.</FONT></TT><BR>
<BR>
In this test implementation, the response is hardcoded short string. For the sake of brevity, I didn't include the test app code, but if necessary I can do so.<BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">I then deployed this </FONT></TT><TT>test </TT><TT><FONT COLOR="#3c3c3c">app to the server, and set up two client machines to load test the server. The load testing code is also written in erlang, and is using ibrowse for making https requests, since I had some problems with httpc. The test client basically adheres to the protocol described above. It spawns request processes, </FONT></TT><TT>with </TT><TT><FONT COLOR="#3c3c3c">each process</FONT></TT><TT> making </TT><TT><FONT COLOR="#3c3c3c">the requests in </FONT></TT><TT>an </TT><TT><FONT COLOR="#3c3c3c">infinite loop. I gather success/failure stats in a</FONT></TT><TT> </TT><TT><FONT COLOR="#3c3c3c">s</FONT></TT><TT>ingleton</TT><TT><FONT COLOR="#3c3c3c"> erlang process and print them out in regular intervals.</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">As in the production system, with http, the </FONT></TT><TT>test </TT><TT><FONT COLOR="#3c3c3c">server performs correctly. With https, when number of concurrent requests reaches some treshold (about 1000), many requests are not served.</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">On the client machine, I occasionally receive following errors:</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">=ERROR REPORT==== </FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">** State machine <0.17592.0> terminating</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">** Reason for termination = </FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">** {badarg,[{erlang,byte_size,[undefined]},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_tls1,split_secret,1},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_tls1,prf,4},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_handshake,master_secret,4},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_connection,handle_resumed_session,2},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_connection,next_state,3},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {gen_fsm,handle_msg,7},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {proc_lib,init_p_do_apply,3}]}</FONT></TT><BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">While on the server I occasionally notice following errors: </FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">=SUPERVISOR REPORT==== 5-May-2011::11:06:33 ===</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> Supervisor: {local,ssl_connection_sup}</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> Context: child_terminated</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> Reason: {{badmatch,{resumed,undefined}},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> [{ssl_handshake,hello,4},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_connection,hello,2},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_connection,next_state,3},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {gen_fsm,handle_msg,7},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {proc_lib,init_p_do_apply,3}]}</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> Offender: [{pid,<0.7044.2>},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {name,undefined},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {mfargs,{ssl_connection,start_link,undefined}},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {restart_type,temporary},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {shutdown,4000},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {child_type,worker}]</FONT></TT><BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">I have tried to tweak some OS parameters, as well as erlang parameters, but without success. After three days of experimenting, I am out of ideas and need some help or pointers.</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">I am using Erlang R14B02 on 64 bit Ubuntu</FONT></TT><TT> 10.04</TT><TT><FONT COLOR="#3c3c3c">, all the machines are on the EC2 cloud. </FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">Following are sysctl.conf settings (I tried some more combinations without success):</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.core.rmem_max = 16777216</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.core.wmem_max = 16777216</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.tcp_rmem = 4096 87380 16777216</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.tcp_wmem = 4096 65536 16777216</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.tcp_syncookies = 1</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.tcp_mem = 50576 64768 98152</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.core.netdev_max_backlog = 2500</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.netfilter.ip_conntrack_max = 1048576</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.ip_local_port_range = 1024 65535</FONT></TT><BR>
<BR>
<TT><FONT COLOR="#3c3c3c">net.ipv4.tcp_fin_timeout = 10</FONT></TT><BR>
<BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">Following is ulimit -a output:</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">core file size (blocks, -c) 0</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">data seg size (kbytes, -d) unlimited</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">scheduling priority (-e) 20</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">file size (blocks, -f) unlimited</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">pending signals (-i) 16382</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">max locked memory (kbytes, -l) 64</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">max memory size (kbytes, -m) unlimited</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">open files (-n) 32000</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">pipe size (512 bytes, -p) 8</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">POSIX message queues (bytes, -q) 819200</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">real-time priority (-r) 0</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">stack size (kbytes, -s) 8192</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">cpu time (seconds, -t) unlimited</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">max user processes (-u) unlimited</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">virtual memory (kbytes, -v) unlimited</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">file locks (-x) unlimited</FONT></TT><BR>
<BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">These are mochiweb parameters:</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">[</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {port, 9999},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {name, https_test},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl, true},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {ssl_opts, [</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {cacertfile, "keys/cacert.pem"},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {certfile, "keys/cert.pem"},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {keyfile, "keys/cert.key"},</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> {depth, 0}</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c"> ]}</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">]</FONT></TT><BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">To use it on a 443 port, I </FONT></TT><TT>have set up iptables port forwarding</TT><BR>
<TT><FONT COLOR="#3c3c3c">iptables -t nat -I PREROUTING --source 0/0 --destination 0/0 -p tcp --dport 443 -j REDIRECT --to-ports 9999</FONT></TT><BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">These are erl parameters:</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">erl -P 268435456 -env ERL_MAX_PORTS 100000</FONT></TT><BR>
<BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">If anybody has any idea, I would be most grateful.</FONT></TT><BR>
<BR>
<BR>
<TT><FONT COLOR="#3c3c3c">Best regards,</FONT></TT><BR>
<TT><FONT COLOR="#3c3c3c">Sasa</FONT></TT>
</BODY>
</HTML>