Hello,<div><br></div><div>I spent last week fighting with SSL problems without success. Eventually I switched to the stunnel based SSL which delegates to Erlang HTTP server.</div><div><br></div><div>Following are the characteristics of the behavior I experienced last week. These are a bit different than the original ones, but I didn't want to open new topic. I am posting this in case someone is stuck with the similar behavior.</div>
<div><br></div><div>1. The mochiweb http server runs without problems since the beginning of the year. It also servers some small number of SSL users. The server is the same erlang application running on the same node.</div>
<div>2. Every day of the last week, in the morning, I would force the users to use SSL (https) rather than http.</div><div>3. The server then works fine for couple of hours.</div><div>4. At one point, the servers starts responding very slowly. Restart of the erlang application, node, or even the machine doesn't help.</div>
<div>5. At the same time, the http server (which is the same application on the same node) is working fine.</div><div>6. Kernel tuning didn't help.</div><div>7. I don't notice anything strange in erlang log or on the OS level.</div>
<div>8. In my application log, it seems as if most connections are not even established. The same thing is confirmed from netstat output. The number of connections is smaller than expected. It looks as if new requests are queued somewhere or some main accept/listen loop is processing them slowly.</div>
<div>9. The CPU usage is low as well as the memory usage.</div><div><br></div><div>After forcing users back to http, the problems disappeared. The server could then normally serve smaller number of SSL users.</div><div><br>
</div><div>It seems to me that the problem is somewhere in the mochiweb/erlang. After switching to stunnel, which is in front of that same mochiweb application, everything is working nicely. I should note that stunnel uses more CPU.</div>
<div><br></div><div>I am not quite sure why is it not working for me. I had load tests which performed fine. Maybe the problem is that my real users are constantly connecting/disconnecting, closing browsers during the request etc. I didn't load test this type of users.</div>
<div>In any case, the behavior described above occurred every day, and was resolved when switching to stunnel SSL.</div><div><br></div><div>I am sorry it didn't work out for me. I would be happier to use erlang SSL, so I'll try again when the new version is released.</div>
<div><br></div><div>Regards,</div><div>Sasa<br><br><div class="gmail_quote">On Fri, May 6, 2011 at 12:02 AM, sasa <span dir="ltr"><<a href="mailto:sasa555@gmail.com">sasa555@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div>
<tt><font color="#3c3c3c">Hello,</font></tt><br>
<br>
First of all, I apologize if this question occurs twice. I have tried submitting it couple of hours ago via google groups, but it didn't show up, so I'm trying via e-mail now.<br>
<br>
<tt><font color="#3c3c3c">I am trying to develop https comet (long polling) server using mochiweb. I have already written one such http server which is in production</font></tt><tt> </tt><tt><font color="#3c3c3c">f</font></tt><tt>or </tt><tt><font color="#3c3c3c">c</font></tt><tt>ouple </tt><tt><font color="#3c3c3c">o</font></tt><tt>f </tt><tt><font color="#3c3c3c">m</font></tt><tt>onths</tt><tt><font color="#3c3c3c"> and serves no more than 1000 concurrent users, sending broadcast message to all concurrent users at an average rate of 0.3 broadcast/sec</font></tt><br>
<br>
<tt><font color="#3c3c3c">When I added ssl support, it essentially works when smaller number of users (about 100-200) are connected. With larger number, it stops responding correctly i.e. some connection attempts are refused, requests time out etc.</font></tt><tt> Only ssl/https part seems to be affected by this behavior.</tt><br>
<br>
<tt><font color="#3c3c3c">To better test this, I developed small test mochiweb app which works as follows:</font></tt><br>
<tt><font color="#3c3c3c">1. Spawn and register singleton broadcast process</font></tt><br>
<tt><font color="#3c3c3c">2. For each request, the erlang process sends its pid to the broadcast process. Then it waits for the message from the broadcast process</font></tt><br>
<tt><font color="#3c3c3c">3. Every 5 seconds broadcast process notifies all registered https response processes from 2, then it clears its list of registered processes</font></tt><br>
<tt><font color="#3c3c3c">4. When process from 2 receives notification, it responds and finishes.</font></tt><br>
<br>
In this test implementation, the response is hardcoded short string. For the sake of brevity, I didn't include the test app code, but if necessary I can do so.<br>
<br>
<br>
<tt><font color="#3c3c3c">I then deployed this </font></tt><tt>test </tt><tt><font color="#3c3c3c">app to the server, and set up two client machines to load test the server. The load testing code is also written in erlang, and is using ibrowse for making https requests, since I had some problems with httpc. The test client basically adheres to the protocol described above. It spawns request processes, </font></tt><tt>with </tt><tt><font color="#3c3c3c">each process</font></tt><tt> making </tt><tt><font color="#3c3c3c">the requests in </font></tt><tt>an </tt><tt><font color="#3c3c3c">infinite loop. I gather success/failure stats in a</font></tt><tt> </tt><tt><font color="#3c3c3c">s</font></tt><tt>ingleton</tt><tt><font color="#3c3c3c"> erlang process and print them out in regular intervals.</font></tt><br>
<br>
<tt><font color="#3c3c3c">As in the production system, with http, the </font></tt><tt>test </tt><tt><font color="#3c3c3c">server performs correctly. With https, when number of concurrent requests reaches some treshold (about 1000), many requests are not served.</font></tt><br>
<br>
<tt><font color="#3c3c3c">On the client machine, I occasionally receive following errors:</font></tt><br>
<tt><font color="#3c3c3c">=ERROR REPORT==== </font></tt><br>
<tt><font color="#3c3c3c">** State machine <0.17592.0> terminating</font></tt><br>
<br>
<tt><font color="#3c3c3c">** Reason for termination = </font></tt><br>
<tt><font color="#3c3c3c">** {badarg,[{erlang,byte_size,[undefined]},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_tls1,split_secret,1},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_tls1,prf,4},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_handshake,master_secret,4},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_connection,handle_resumed_session,2},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_connection,next_state,3},</font></tt><br>
<tt><font color="#3c3c3c"> {gen_fsm,handle_msg,7},</font></tt><br>
<tt><font color="#3c3c3c"> {proc_lib,init_p_do_apply,3}]}</font></tt><br>
<br>
<br>
<tt><font color="#3c3c3c">While on the server I occasionally notice following errors: </font></tt><br>
<tt><font color="#3c3c3c">=SUPERVISOR REPORT==== 5-May-2011::11:06:33 ===</font></tt><br>
<tt><font color="#3c3c3c"> Supervisor: {local,ssl_connection_sup}</font></tt><br>
<tt><font color="#3c3c3c"> Context: child_terminated</font></tt><br>
<tt><font color="#3c3c3c"> Reason: {{badmatch,{resumed,undefined}},</font></tt><br>
<tt><font color="#3c3c3c"> [{ssl_handshake,hello,4},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_connection,hello,2},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_connection,next_state,3},</font></tt><br>
<tt><font color="#3c3c3c"> {gen_fsm,handle_msg,7},</font></tt><br>
<tt><font color="#3c3c3c"> {proc_lib,init_p_do_apply,3}]}</font></tt><br>
<tt><font color="#3c3c3c"> Offender: [{pid,<0.7044.2>},</font></tt><br>
<tt><font color="#3c3c3c"> {name,undefined},</font></tt><br>
<tt><font color="#3c3c3c"> {mfargs,{ssl_connection,start_link,undefined}},</font></tt><br>
<tt><font color="#3c3c3c"> {restart_type,temporary},</font></tt><br>
<tt><font color="#3c3c3c"> {shutdown,4000},</font></tt><br>
<tt><font color="#3c3c3c"> {child_type,worker}]</font></tt><br>
<br>
<br>
<tt><font color="#3c3c3c">I have tried to tweak some OS parameters, as well as erlang parameters, but without success. After three days of experimenting, I am out of ideas and need some help or pointers.</font></tt><br>
<br>
<tt><font color="#3c3c3c">I am using Erlang R14B02 on 64 bit Ubuntu</font></tt><tt> 10.04</tt><tt><font color="#3c3c3c">, all the machines are on the EC2 cloud. </font></tt><br>
<br>
<tt><font color="#3c3c3c">Following are sysctl.conf settings (I tried some more combinations without success):</font></tt><br>
<tt><font color="#3c3c3c">net.core.rmem_max = 16777216</font></tt><br>
<tt><font color="#3c3c3c">net.core.wmem_max = 16777216</font></tt><br>
<tt><font color="#3c3c3c">net.ipv4.tcp_rmem = 4096 87380 16777216</font></tt><br>
<tt><font color="#3c3c3c">net.ipv4.tcp_wmem = 4096 65536 16777216</font></tt><br>
<tt><font color="#3c3c3c">net.ipv4.tcp_syncookies = 1</font></tt><br>
<br>
<tt><font color="#3c3c3c">net.ipv4.tcp_mem = 50576 64768 98152</font></tt><br>
<tt><font color="#3c3c3c">net.core.netdev_max_backlog = 2500</font></tt><br>
<tt><font color="#3c3c3c">net.ipv4.netfilter.ip_conntrack_max = 1048576</font></tt><br>
<br>
<tt><font color="#3c3c3c">net.ipv4.ip_local_port_range = 1024 65535</font></tt><br>
<br>
<tt><font color="#3c3c3c">net.ipv4.tcp_fin_timeout = 10</font></tt><br>
<br>
<br>
<br>
<tt><font color="#3c3c3c">Following is ulimit -a output:</font></tt><br>
<tt><font color="#3c3c3c">core file size (blocks, -c) 0</font></tt><br>
<tt><font color="#3c3c3c">data seg size (kbytes, -d) unlimited</font></tt><br>
<tt><font color="#3c3c3c">scheduling priority (-e) 20</font></tt><br>
<tt><font color="#3c3c3c">file size (blocks, -f) unlimited</font></tt><br>
<tt><font color="#3c3c3c">pending signals (-i) 16382</font></tt><br>
<tt><font color="#3c3c3c">max locked memory (kbytes, -l) 64</font></tt><br>
<tt><font color="#3c3c3c">max memory size (kbytes, -m) unlimited</font></tt><br>
<tt><font color="#3c3c3c">open files (-n) 32000</font></tt><br>
<tt><font color="#3c3c3c">pipe size (512 bytes, -p) 8</font></tt><br>
<tt><font color="#3c3c3c">POSIX message queues (bytes, -q) 819200</font></tt><br>
<tt><font color="#3c3c3c">real-time priority (-r) 0</font></tt><br>
<tt><font color="#3c3c3c">stack size (kbytes, -s) 8192</font></tt><br>
<tt><font color="#3c3c3c">cpu time (seconds, -t) unlimited</font></tt><br>
<tt><font color="#3c3c3c">max user processes (-u) unlimited</font></tt><br>
<tt><font color="#3c3c3c">virtual memory (kbytes, -v) unlimited</font></tt><br>
<tt><font color="#3c3c3c">file locks (-x) unlimited</font></tt><br>
<br>
<br>
<br>
<tt><font color="#3c3c3c">These are mochiweb parameters:</font></tt><br>
<tt><font color="#3c3c3c">[</font></tt><br>
<tt><font color="#3c3c3c"> {port, 9999},</font></tt><br>
<tt><font color="#3c3c3c"> {name, https_test},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl, true},</font></tt><br>
<tt><font color="#3c3c3c"> {ssl_opts, [</font></tt><br>
<tt><font color="#3c3c3c"> {cacertfile, "keys/cacert.pem"},</font></tt><br>
<tt><font color="#3c3c3c"> {certfile, "keys/cert.pem"},</font></tt><br>
<tt><font color="#3c3c3c"> {keyfile, "keys/cert.key"},</font></tt><br>
<tt><font color="#3c3c3c"> {depth, 0}</font></tt><br>
<tt><font color="#3c3c3c"> ]}</font></tt><br>
<tt><font color="#3c3c3c">]</font></tt><br>
<br>
<br>
<tt><font color="#3c3c3c">To use it on a 443 port, I </font></tt><tt>have set up iptables port forwarding</tt><br>
<tt><font color="#3c3c3c">iptables -t nat -I PREROUTING --source 0/0 --destination 0/0 -p tcp --dport 443 -j REDIRECT --to-ports 9999</font></tt><br>
<br>
<br>
<tt><font color="#3c3c3c">These are erl parameters:</font></tt><br>
<tt><font color="#3c3c3c">erl -P 268435456 -env ERL_MAX_PORTS 100000</font></tt><br>
<br>
<br>
<br>
<tt><font color="#3c3c3c">If anybody has any idea, I would be most grateful.</font></tt><br>
<br>
<br>
<tt><font color="#3c3c3c">Best regards,</font></tt><br><font color="#888888">
<tt><font color="#3c3c3c">Sasa</font></tt>
</font></div>
</blockquote></div><br></div>