<div dir="ltr">In resolution, it appears that the only way we have been able to resolve our issue was to stop reusing ssl sessions. With Ingeal's great help we were able to track-down several symptoms. (Many connections being started resulting in the ssl_otp_session_cache ets table growing quite large and the ets lookup to find the best session to reuse taking ~45msec each time.) By removing session reuse we are forcing re-negotiation on each connection but this has proven to still be faster than incurring the ets table lookup.<div>
<br></div><div>Thank you very much Ingela. Hoepfully we can figure-out a better long-term solution in the near future.</div><div><br></div><div>Best,</div><div><br></div><div>Ryan Brown</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Sep 26, 2013 at 1:51 AM, Ingela Andin <span dir="ltr"><<a href="mailto:ingela@andin.se" target="_blank">ingela@andin.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Ryan!<br><div><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On Wed, Sep 25, 2013 at 11:28 PM, Ryan Brown <span dir="ltr"><<a href="mailto:ryankbrown@gmail.com" target="_blank">ryankbrown@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you Ingela. I will work on creating a test to compare to openssl. <div><br></div>
</div></blockquote></div><div><br><div>When you do the test with openssl make sure that you run the same cipher suit as when you run erlang, and<br></div>the same version of the TLS protocol. <br><br> </div><div class="im">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><div>In the mean time, what we are currently doing is this. Our application sends requests to various hosts via ibrowse (http/s). The messages are basic HTTP form POSTs (application/x-www-form-urlencoded). Our test just sends these messages to our application which pops them on a queue in RabbitMQ then one-by-one pulls them off and attempt to deliver them. Fairly straightforward really.</div>
<div><br></div><br></div></blockquote><div><br></div></div><div>But so many options and things that can be going on in under the covers ... I will help you to look into it.<br><br><br></div><div>Regards Ingela Erlang/OTP - team Ericsson AB<br>
</div><div><div class="h5"><div><br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div class="gmail_quote">On Wed, Sep 25, 2013 at 2:59 PM, Ingela Andin <span dir="ltr"><<a href="mailto:ingela@andin.se" target="_blank">ingela@andin.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi!<br><div class="gmail_extra"><div><br>On Wed, Sep 25, 2013 at 7:07 PM, Ryan Brown <span dir="ltr"><<a href="mailto:ryankbrown@gmail.com" target="_blank">ryankbrown@gmail.com</a>></span> wrote:<br>
</div><div class="gmail_quote"><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Excellent Ingela! By upgrading to the R!6B02 we have been able to eliminate the above stated issue with the Erlang message queues growing. However, we are still seeing VERY slow throughput to ssl endpoints. In our limited, scaled-back, test. We are seeing message throughput to an ssl endpoint at 12 msgs/sec and non-ssl of 300+ msgs/sec. Is there a connection limit for ssl connections related to Erlang directly?<div>
<br></div><div>Thank you!</div><div><br></div><div><br></div></div></blockquote><div><br></div></div><div>Could you send me more details how you are performing this test. It would be interesting if I could run the same kind of test as you do. And when you say messages what more exactly do mean? Messages can be on many abstraction levels (tcp messages, tls messages, http messages etc). Could you perform</div>
<div>the same kind of throughput test running for instance openssl?</div><div><div><br></div><div>Regards Ingela Erlang/OTP team - Ericsson AB </div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail_extra"><div><br><div class="gmail_quote">On Wed, Sep 25, 2013 at 6:58 AM, Ingela Andin <span dir="ltr"><<a href="mailto:ingela.andin@gmail.com" target="_blank">ingela.andin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi!<br><div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>2013/9/25 Ryan Brown <span dir="ltr"><<a href="mailto:ryankbrown@gmail.com" target="_blank">ryankbrown@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div><span style="font-family:arial,sans-serif;font-size:13px">Hi Ingela,</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
Thank you for the response. I suppose I don't really know how to see what messages are in the ssl queue. What I am looking at is nregs and watching the "msgs" column growing at a steady pace. Is there a way to look at these messages?</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div></div></div></blockquote></div><div><br><div>You can use the observer application to look at erlang processes.<br><br><a href="http://www.erlang.org/erldoc?q=observe&x=0&y=0" target="_blank">http://www.erlang.org/erldoc?q=observe&x=0&y=0</a><br>
</div><div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="font-family:arial,sans-serif;font-size:13px">
As
for reproducing in the latest version of ssl. I was planning on
upgrading a machine to 16 today and attempting to reproduce. This should
get me the latest ssl version correct?</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div></div></div></blockquote><div><br></div></div><div>R16B02
(ssl-5.3.1) is the latest. There is a bug in ssl-5.1 (R15B02) version (fix in ssl5.2) that
perhaps could be related to your problem. <br><br></div><div>Regards Ingela Erlang/OTP Team - Ericsson AB<br></div><div><div><br></div><br><div class="gmail_quote">On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin <span dir="ltr"><<a href="mailto:ingela.andin@gmail.com" target="_blank">ingela.andin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra">Hi!<br></div><div class="gmail_extra"><br><div class="gmail_quote">
<div>
2013/9/24 Ryan Brown <span dir="ltr"><<a href="mailto:ryankbrown@gmail.com" target="_blank">ryankbrown@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">We have a fairly high-volume (28m+ messages/day) application that works, essentially, as a pub-sub for many internal and external applications. We are experiencing an issue under load where and subscribers that are using ssl are experiencing a significant slowness in delivery. When we restart the server we begin watching the processes in our application and we can, very reliably and fairly quickly, see the ssl_manager_sup process message queue start increasing in size. As this happens, all messages for those subscribers with ssl endpoints begin to back-up and delivery slows to a near halt.<div>
<br></div></div></blockquote></div><div><div><br>What messages are there in the queue?<br> <br><br></div></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>It appears that ssl is a bit of a known issue within erlang. Or, at least, a tricky use-case to perfect. Are there any known issues that we can address in terms of configuration, best practices in setting-up ssl connections or anything else for that matter we can be doing to resolve this issue or more clearly identify the root cause? We're in active troubleshooting mode and doing what we can to dig deeper including dissecting the code to the kernel.</div>
<div><br></div><div>We are running Erlang 15B02 (64bit).</div><div><br></div><br></div></blockquote></div></div><div><br><div>So that would be ssl-5.1? Is it possible to reproduce the problem on the latest version ?<br>
<br></div>Regards Ingela Erlang/OTP team - Ericsson AB</div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra">
<span><font color="#888888">-- <br>-rb
</font></span></div></div>
</blockquote></font></span></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div><span><font color="#888888">-- <br>-rb
</font></span></div>
</blockquote></div></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div><div class="gmail_extra"><span><font color="#888888">-- <br>-rb
</font></span></div>
</blockquote></div></div></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>-rb
</div>