[erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory
Sanath Prasanna
ahe.sanath@REDACTED
Mon Sep 28 13:42:27 CEST 2015
Hi all,
Any update on this?
Br,
Robert
On Fri, Sep 18, 2015 at 2:11 PM, Eranga Udesh <eranga.erl@REDACTED> wrote:
> Hi,
>
> I am experiencing the same issue in,
>
> - Erlang version : V6.4 , (Erlang/OTP 17 [erts-6.4] [source] [64-bit])
> - OS version/architecture (32/64 bit) : Red Hat Enterprise Linux
> Server release 6.6 (64 bit)
>
> The issue doesn't come always but after running the system for 3-10 days
> can experience the tls_connection instances making high reductions and
> consuming high memory as sent by Sanath. Since it's a production system, I
> wasn't able to do much experiments, but applied the patches you sent, which
> didn't solve this issue.
>
> There are 2 servers running Erlang VMs making SSL requests to the same
> HTTPS host. When this issue happens, it happens in both the VMs. So it
> could be triggered by some conditions in SSL or network connection. Even
> though I left the VMs to run for a while to see if they can recover, but no
> success. However if I restart the VMs, it starts to run as normal.
>
> So the summary is,
>
> - It triggers by some condition in remote SSL host or network
> connection
> - Erlang VMs don't recover itself
> - Once restarted, it start working, which implies the issue is local.
> i.e. in Erlang VM (tls_connection)
>
> We will try to recreate it and send you more details. I wonder if anybody
> else is experiencing such issue.
>
> Tks,
> - Eranga
>
>
>
>
>
>
> On Fri, Sep 18, 2015 at 1:36 PM, Ingela Andin <ingela.andin@REDACTED>
> wrote:
>
>> Hi!
>>
>> As we are very busy with the release 18.1 I have not had time to try and
>> recreate your problem. What version of OTP and the ssl application are you
>> using?
>> Can you reproduce the problem with the latest on github?
>>
>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>
>>
>> On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>> wrote:
>>
>>> Hi Ingela,
>>> Any update on this??
>>> "Even apply your patch, *still problem is persist.* Any
>>> more suggestions to solve this unexpected behavior ? "
>>> Br,
>>> Robert
>>>
>>> On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>> wrote:
>>>
>>>> Hi Ingela,
>>>> Even apply your patch, *still problem is persist.* Any
>>>> more suggestions to solve this unexpected behavior ?
>>>> Br,
>>>> Robert
>>>>
>>>> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi Ingela.
>>>>> Tx a lot for your help & patch related to that.I'll inform you the
>>>>> result after applying & testing patch.
>>>>> Br,
>>>>> Robert
>>>>>
>>>>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> It could be an bug that in the ssl application that I just fixed. The
>>>>>> default session cache
>>>>>> was violating the API, and this in turn made the mechanism for not
>>>>>> registering a lot of equivalent
>>>>>> sessions in the client fail.
>>>>>>
>>>>>> Here is the patch:
>>>>>>
>>>>>>
>>>>>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl
>>>>>> index 1770faf..0d6cc93 100644
>>>>>> --- a/lib/ssl/src/ssl_session.erl
>>>>>> +++ b/lib/ssl/src/ssl_session.erl
>>>>>> @@ -100,14 +100,14 @@ select_session([], _, _) ->
>>>>>> no_session;
>>>>>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) ->
>>>>>> IsNotResumable =
>>>>>> - fun([_Id, Session]) ->
>>>>>> + fun(Session) ->
>>>>>> not (resumable(Session#session.is_resumable) andalso
>>>>>> lists:member(Session#session.cipher_suite, Ciphers)
>>>>>> andalso (OwnCert == Session#session.own_certificate))
>>>>>> end,
>>>>>> case lists:dropwhile(IsNotResumable, Sessions) of
>>>>>> [] -> no_session;
>>>>>> - [[Id, _]|_] -> Id
>>>>>> + [Session | _] -> Session#session.session_id
>>>>>> end.
>>>>>>
>>>>>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _)
>>>>>> ->
>>>>>> diff --git a/lib/ssl/src/ssl_session_cache.erl
>>>>>> b/lib/ssl/src/ssl_session_cach
>>>>>> e.erl
>>>>>> index 11ed310..cfc48cd 100644
>>>>>> --- a/lib/ssl/src/ssl_session_cache.erl
>>>>>> +++ b/lib/ssl/src/ssl_session_cache.erl
>>>>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) ->
>>>>>>
>>>>>> %%--------------------------------------------------------------------
>>>>>> select_session(Cache, PartialKey) ->
>>>>>> ets:select(Cache,
>>>>>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]).
>>>>>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]).
>>>>>>
>>>>>>
>>>>>> %%--------------------------------------------------------------------
>>>>>> %%% Internal functions
>>>>>>
>>>>>>
>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>
>>>>>>
>>>>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna <ahe.sanath@REDACTED>:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I am running HTTP client using httpc Module to send both http and
>>>>>>> https requests. normally sending arround 300 request per second without any
>>>>>>> issue. however sometimes erlang node become very slow responsive. at that
>>>>>>> time server load average is very high and using etop can identify
>>>>>>> "tls_connection" process take more memory. when restart the erlang node its
>>>>>>> become normal. as per my investigation normal time memory, processors,
>>>>>>> loadAverage is not increasing. following is the HTTP request config
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync,
>>>>>>> false}])
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> below is the etop output at that time
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> procs 1134 processes 1504844
>>>>>>> code 9309
>>>>>>>
>>>>>>> runq 0 atom 420
>>>>>>> ets 29692
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Pid Name or Initial Func Time Reds Memory MsgQ
>>>>>>> Current Function
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------------------------------
>>>>>>>
>>>>>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0
>>>>>>> gen_fsm:loop/7
>>>>>>>
>>>>>>>
>>>>>>> can some one help me to solve this issue?
>>>>>>>
>>>>>>> Br,
>>>>>>>
>>>>>>> A.H.E. Robert
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> erlang-questions mailing list
>>>>>>> erlang-questions@REDACTED
>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150928/ddf8e9f3/attachment.htm>
More information about the erlang-questions
mailing list