[erlang-questions] Problematic SSL message publication

[Ryan Brown]

In resolution, it appears that the only way we have been able to resolve
our issue was to stop reusing ssl sessions. With Ingeal's great help we
were able to track-down several symptoms. (Many connections being started
resulting in the ssl_otp_session_cache ets table growing quite large and
the ets lookup to find the best session to reuse taking ~45msec each time.)
By removing session reuse we are forcing re-negotiation on each connection
but this has proven to still be faster than incurring the ets table lookup.

Thank you very much Ingela. Hoepfully we can figure-out a better long-term
solution in the near future.

Best,

Ryan Brown


On Thu, Sep 26, 2013 at 1:51 AM, Ingela Andin <ingela@REDACTED> wrote:

> Hi Ryan!
>
>
> On Wed, Sep 25, 2013 at 11:28 PM, Ryan Brown <ryankbrown@REDACTED> wrote:
>
>> Thank you Ingela. I will work on creating a test to compare to openssl.
>>
>>
> When you do the test with openssl make sure that you run the same cipher
> suit as when you run erlang, and
> the same version of the TLS protocol.
>
>
>
>> In the mean time, what we are currently doing is this. Our application
>> sends requests to various hosts via ibrowse (http/s). The messages are
>> basic HTTP  form POSTs (application/x-www-form-urlencoded). Our test just
>> sends these messages to our application which pops them on a queue in
>> RabbitMQ then one-by-one pulls them off and attempt to deliver them. Fairly
>> straightforward really.
>>
>>
>>
> But so many options and things that can be going on in under the covers
> ... I will help you to look into it.
>
>
> Regards Ingela Erlang/OTP - team Ericsson AB
>
>
>
>
>>
>> On Wed, Sep 25, 2013 at 2:59 PM, Ingela Andin <ingela@REDACTED> wrote:
>>
>>> Hi!
>>>
>>> On Wed, Sep 25, 2013 at 7:07 PM, Ryan Brown <ryankbrown@REDACTED>wrote:
>>>
>>>> Excellent Ingela! By upgrading to the R!6B02 we have been able to
>>>> eliminate the above stated issue with the Erlang message queues growing.
>>>> However, we are still seeing VERY slow throughput to ssl endpoints. In our
>>>> limited, scaled-back, test. We are seeing message throughput to an ssl
>>>> endpoint at 12 msgs/sec and non-ssl of 300+ msgs/sec. Is there a connection
>>>> limit for ssl connections related to Erlang directly?
>>>>
>>>> Thank you!
>>>>
>>>>
>>>>
>>> Could you send me more details how you are performing this test. It
>>> would be interesting if I could run the same kind of test as you do. And
>>> when you say messages what more exactly do mean? Messages can be on many
>>> abstraction levels (tcp messages, tls messages, http messages etc).  Could
>>> you perform
>>> the same kind of throughput test running for instance openssl?
>>>
>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>
>>>
>>>
>>>
>>>>
>>>> On Wed, Sep 25, 2013 at 6:58 AM, Ingela Andin <ingela.andin@REDACTED>wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>>
>>>>> 2013/9/25 Ryan Brown <ryankbrown@REDACTED>
>>>>>
>>>>>> Hi Ingela,
>>>>>>
>>>>>> Thank you for the response. I suppose I don't really know how to see
>>>>>> what messages are in the ssl queue. What I am looking at is nregs and
>>>>>> watching the "msgs" column growing at a steady pace. Is there a way to look
>>>>>> at these messages?
>>>>>>
>>>>>>
>>>>> You can use the observer application to look at erlang processes.
>>>>>
>>>>> http://www.erlang.org/erldoc?q=observe&x=0&y=0
>>>>>
>>>>>
>>>>>
>>>>>> As for reproducing in the latest version of ssl. I was planning on
>>>>>> upgrading a machine to 16 today and attempting to reproduce. This should
>>>>>> get me the latest ssl version correct?
>>>>>>
>>>>>>
>>>>> R16B02  (ssl-5.3.1) is the latest. There is a bug in ssl-5.1 (R15B02)
>>>>> version (fix in ssl5.2) that perhaps could be related to your problem.
>>>>>
>>>>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>>>>
>>>>>
>>>>> On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin <ingela.andin@REDACTED>wrote:
>>>>>
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>>  2013/9/24 Ryan Brown <ryankbrown@REDACTED>
>>>>>>
>>>>>>> We have a fairly high-volume (28m+ messages/day) application that
>>>>>>> works, essentially, as a pub-sub for many internal and external
>>>>>>> applications. We are experiencing an issue under load where and subscribers
>>>>>>> that are using ssl are experiencing a significant slowness in delivery.
>>>>>>> When we restart the server we begin watching the processes in our
>>>>>>> application and we can, very reliably and fairly quickly, see the
>>>>>>> ssl_manager_sup process message queue start increasing in size. As this
>>>>>>> happens, all messages for those subscribers with ssl endpoints begin to
>>>>>>> back-up and delivery slows to a near halt.
>>>>>>>
>>>>>>>
>>>>>> What messages are there in the queue?
>>>>>>
>>>>>>
>>>>>>  It appears that ssl is a bit of a known issue within erlang. Or, at
>>>>>>> least, a tricky use-case to perfect. Are there any known issues that we can
>>>>>>> address in terms of configuration, best practices in setting-up ssl
>>>>>>> connections or anything else for that matter we can be doing to resolve
>>>>>>> this issue or more clearly identify the root cause? We're in active
>>>>>>> troubleshooting mode and doing what we can to dig deeper including
>>>>>>> dissecting the code to the kernel.
>>>>>>>
>>>>>>> We are running Erlang 15B02 (64bit).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> So that would be ssl-5.1?  Is it possible to reproduce the problem on
>>>>>> the latest version ?
>>>>>>
>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>>> -rb
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -rb
>>>>
>>>
>>>
>>
>>
>> --
>> -rb
>>
>
>


-- 
-rb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20131004/16043acc/attachment.htm>