[erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory

Sanath Prasanna ahe.sanath@REDACTED
Mon Sep 28 13:42:27 CEST 2015


Hi all,
Any update on this?
Br,
Robert

On Fri, Sep 18, 2015 at 2:11 PM, Eranga Udesh <eranga.erl@REDACTED> wrote:

> Hi,
>
> I am experiencing the same issue in,
>
>    - Erlang version : V6.4 , (Erlang/OTP 17 [erts-6.4] [source] [64-bit])
>    - OS version/architecture (32/64 bit) : Red Hat Enterprise Linux
>    Server release 6.6 (64 bit)
>
> The issue doesn't come always but after running the system for 3-10 days
> can experience the tls_connection instances making high reductions and
> consuming high memory as sent by Sanath. Since it's a production system, I
> wasn't able to do much experiments, but applied the patches you sent, which
> didn't solve this issue.
>
> There are 2 servers running Erlang VMs making SSL requests to the same
> HTTPS host. When this issue happens, it happens in both the VMs. So it
> could be triggered by some conditions in SSL or network connection. Even
> though I left the VMs to run for a while to see if they can recover, but no
> success. However if I restart the VMs, it starts to run as normal.
>
> So the summary is,
>
>    - It triggers by some condition in remote SSL host or network
>    connection
>    - Erlang VMs don't recover itself
>    - Once restarted, it start working, which implies the issue is local.
>    i.e. in Erlang VM (tls_connection)
>
> We will try to recreate it and send you more details. I wonder if anybody
> else is experiencing such issue.
>
> Tks,
> - Eranga
>
>
>
>
>
>
> On Fri, Sep 18, 2015 at 1:36 PM, Ingela Andin <ingela.andin@REDACTED>
> wrote:
>
>> Hi!
>>
>> As we are very busy with the release 18.1 I have not had time to try and
>> recreate your problem. What version of OTP  and the ssl application are you
>> using?
>> Can you reproduce the problem with the latest on github?
>>
>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>
>>
>> On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>> wrote:
>>
>>> Hi Ingela,
>>> Any update on this??
>>> "Even apply your patch, *still problem is persist.* Any
>>> more suggestions to solve this unexpected behavior ? "
>>> Br,
>>> Robert
>>>
>>> On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>> wrote:
>>>
>>>> Hi Ingela,
>>>> Even apply your patch, *still problem is persist.* Any
>>>> more suggestions to solve this unexpected behavior ?
>>>> Br,
>>>> Robert
>>>>
>>>> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi Ingela.
>>>>> Tx a lot for your help & patch related to that.I'll inform you the
>>>>> result after applying &  testing patch.
>>>>> Br,
>>>>> Robert
>>>>>
>>>>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> It could be an bug that in the ssl application that I just fixed. The
>>>>>> default session cache
>>>>>> was violating the API, and this in turn made the mechanism for not
>>>>>> registering a lot of equivalent
>>>>>> sessions in the client fail.
>>>>>>
>>>>>>  Here is the patch:
>>>>>>
>>>>>>
>>>>>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl
>>>>>> index 1770faf..0d6cc93 100644
>>>>>> --- a/lib/ssl/src/ssl_session.erl
>>>>>> +++ b/lib/ssl/src/ssl_session.erl
>>>>>> @@ -100,14 +100,14 @@ select_session([], _, _) ->
>>>>>>      no_session;
>>>>>>  select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) ->
>>>>>>      IsNotResumable =
>>>>>> - fun([_Id, Session]) ->
>>>>>> + fun(Session) ->
>>>>>>   not (resumable(Session#session.is_resumable) andalso
>>>>>>       lists:member(Session#session.cipher_suite, Ciphers)
>>>>>>       andalso (OwnCert == Session#session.own_certificate))
>>>>>>   end,
>>>>>>      case lists:dropwhile(IsNotResumable, Sessions) of
>>>>>>   [] ->   no_session;
>>>>>> - [[Id, _]|_] -> Id
>>>>>> + [Session | _] -> Session#session.session_id
>>>>>>      end.
>>>>>>
>>>>>>  is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _)
>>>>>> ->
>>>>>> diff --git a/lib/ssl/src/ssl_session_cache.erl
>>>>>> b/lib/ssl/src/ssl_session_cach
>>>>>> e.erl
>>>>>> index 11ed310..cfc48cd 100644
>>>>>> --- a/lib/ssl/src/ssl_session_cache.erl
>>>>>> +++ b/lib/ssl/src/ssl_session_cache.erl
>>>>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) ->
>>>>>>
>>>>>>  %%--------------------------------------------------------------------
>>>>>>  select_session(Cache, PartialKey) ->
>>>>>>      ets:select(Cache,
>>>>>> -       [{{{PartialKey,'$1'}, '$2'},[],['$$']}]).
>>>>>> +       [{{{PartialKey,'_'}, '$1'},[],['$1']}]).
>>>>>>
>>>>>>
>>>>>>  %%--------------------------------------------------------------------
>>>>>>  %%% Internal functions
>>>>>>
>>>>>>
>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>
>>>>>>
>>>>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna <ahe.sanath@REDACTED>:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I am running HTTP client using httpc Module to send both http and
>>>>>>> https requests. normally sending arround 300 request per second without any
>>>>>>> issue. however sometimes erlang node become very slow responsive. at that
>>>>>>> time server load average is very high and using etop can identify
>>>>>>> "tls_connection" process take more memory. when restart the erlang node its
>>>>>>> become normal. as per my investigation normal time memory, processors,
>>>>>>> loadAverage is not increasing. following is the HTTP request config
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync,
>>>>>>> false}])
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> below is the etop output at that time
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         procs    1134                        processes 1504844
>>>>>>> code         9309
>>>>>>>
>>>>>>>         runq        0                        atom          420
>>>>>>> ets         29692
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Pid            Name or Initial Func    Time    Reds  Memory    MsgQ
>>>>>>> Current Function
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------------------------------
>>>>>>>
>>>>>>> <5490.26428.14>tls_connection:init/     '-'    733224580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26429.14>tls_connection:init/     '-'   1328524580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26430.14>tls_connection:init/     '-'    528924580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26431.14>tls_connection:init/     '-'   1432224580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26432.14>tls_connection:init/     '-'       024580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26433.14>tls_connection:init/     '-'       024580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26434.14>tls_connection:init/     '-'       024580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26435.14>tls_connection:init/     '-'       024580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26436.14>tls_connection:init/     '-'       024580768       0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>>
>>>>>>> can some one help me to solve this issue?
>>>>>>>
>>>>>>> Br,
>>>>>>>
>>>>>>> A.H.E. Robert
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> erlang-questions mailing list
>>>>>>> erlang-questions@REDACTED
>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150928/ddf8e9f3/attachment.htm>


More information about the erlang-questions mailing list