[erlang-questions] Problematic SSL message publication

Ryan Brown <>
Fri Oct 4 22:31:51 CEST 2013


In resolution, it appears that the only way we have been able to resolve
our issue was to stop reusing ssl sessions. With Ingeal's great help we
were able to track-down several symptoms. (Many connections being started
resulting in the ssl_otp_session_cache ets table growing quite large and
the ets lookup to find the best session to reuse taking ~45msec each time.)
By removing session reuse we are forcing re-negotiation on each connection
but this has proven to still be faster than incurring the ets table lookup.

Thank you very much Ingela. Hoepfully we can figure-out a better long-term
solution in the near future.

Best,

Ryan Brown


On Thu, Sep 26, 2013 at 1:51 AM, Ingela Andin <> wrote:

> Hi Ryan!
>
>
> On Wed, Sep 25, 2013 at 11:28 PM, Ryan Brown <> wrote:
>
>> Thank you Ingela. I will work on creating a test to compare to openssl.
>>
>>
> When you do the test with openssl make sure that you run the same cipher
> suit as when you run erlang, and
> the same version of the TLS protocol.
>
>
>
>> In the mean time, what we are currently doing is this. Our application
>> sends requests to various hosts via ibrowse (http/s). The messages are
>> basic HTTP  form POSTs (application/x-www-form-urlencoded). Our test just
>> sends these messages to our application which pops them on a queue in
>> RabbitMQ then one-by-one pulls them off and attempt to deliver them. Fairly
>> straightforward really.
>>
>>
>>
> But so many options and things that can be going on in under the covers
> ... I will help you to look into it.
>
>
> Regards Ingela Erlang/OTP - team Ericsson AB
>
>
>
>
>>
>> On Wed, Sep 25, 2013 at 2:59 PM, Ingela Andin <> wrote:
>>
>>> Hi!
>>>
>>> On Wed, Sep 25, 2013 at 7:07 PM, Ryan Brown <>wrote:
>>>
>>>> Excellent Ingela! By upgrading to the R!6B02 we have been able to
>>>> eliminate the above stated issue with the Erlang message queues growing.
>>>> However, we are still seeing VERY slow throughput to ssl endpoints. In our
>>>> limited, scaled-back, test. We are seeing message throughput to an ssl
>>>> endpoint at 12 msgs/sec and non-ssl of 300+ msgs/sec. Is there a connection
>>>> limit for ssl connections related to Erlang directly?
>>>>
>>>> Thank you!
>>>>
>>>>
>>>>
>>> Could you send me more details how you are performing this test. It
>>> would be interesting if I could run the same kind of test as you do. And
>>> when you say messages what more exactly do mean? Messages can be on many
>>> abstraction levels (tcp messages, tls messages, http messages etc).  Could
>>> you perform
>>> the same kind of throughput test running for instance openssl?
>>>
>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>
>>>
>>>
>>>
>>>>
>>>> On Wed, Sep 25, 2013 at 6:58 AM, Ingela Andin <>wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>>
>>>>> 2013/9/25 Ryan Brown <>
>>>>>
>>>>>> Hi Ingela,
>>>>>>
>>>>>> Thank you for the response. I suppose I don't really know how to see
>>>>>> what messages are in the ssl queue. What I am looking at is nregs and
>>>>>> watching the "msgs" column growing at a steady pace. Is there a way to look
>>>>>> at these messages?
>>>>>>
>>>>>>
>>>>> You can use the observer application to look at erlang processes.
>>>>>
>>>>> http://www.erlang.org/erldoc?q=observe&x=0&y=0
>>>>>
>>>>>
>>>>>
>>>>>> As for reproducing in the latest version of ssl. I was planning on
>>>>>> upgrading a machine to 16 today and attempting to reproduce. This should
>>>>>> get me the latest ssl version correct?
>>>>>>
>>>>>>
>>>>> R16B02  (ssl-5.3.1) is the latest. There is a bug in ssl-5.1 (R15B02)
>>>>> version (fix in ssl5.2) that perhaps could be related to your problem.
>>>>>
>>>>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>>>>
>>>>>
>>>>> On Wed, Sep 25, 2013 at 2:09 AM, Ingela Andin <>wrote:
>>>>>
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>>  2013/9/24 Ryan Brown <>
>>>>>>
>>>>>>> We have a fairly high-volume (28m+ messages/day) application that
>>>>>>> works, essentially, as a pub-sub for many internal and external
>>>>>>> applications. We are experiencing an issue under load where and subscribers
>>>>>>> that are using ssl are experiencing a significant slowness in delivery.
>>>>>>> When we restart the server we begin watching the processes in our
>>>>>>> application and we can, very reliably and fairly quickly, see the
>>>>>>> ssl_manager_sup process message queue start increasing in size. As this
>>>>>>> happens, all messages for those subscribers with ssl endpoints begin to
>>>>>>> back-up and delivery slows to a near halt.
>>>>>>>
>>>>>>>
>>>>>> What messages are there in the queue?
>>>>>>
>>>>>>
>>>>>>  It appears that ssl is a bit of a known issue within erlang. Or, at
>>>>>>> least, a tricky use-case to perfect. Are there any known issues that we can
>>>>>>> address in terms of configuration, best practices in setting-up ssl
>>>>>>> connections or anything else for that matter we can be doing to resolve
>>>>>>> this issue or more clearly identify the root cause? We're in active
>>>>>>> troubleshooting mode and doing what we can to dig deeper including
>>>>>>> dissecting the code to the kernel.
>>>>>>>
>>>>>>> We are running Erlang 15B02 (64bit).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> So that would be ssl-5.1?  Is it possible to reproduce the problem on
>>>>>> the latest version ?
>>>>>>
>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>>> -rb
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -rb
>>>>
>>>
>>>
>>
>>
>> --
>> -rb
>>
>
>


-- 
-rb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20131004/16043acc/attachment.html>


More information about the erlang-questions mailing list