[erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory
Sanath Prasanna
ahe.sanath@REDACTED
Thu Oct 1 11:01:25 CEST 2015
Hi all,
Any update on this?
Br,
Robert
On Mon, Sep 28, 2015 at 5:12 PM, Sanath Prasanna <ahe.sanath@REDACTED>
wrote:
> Hi all,
> Any update on this?
> Br,
> Robert
>
> On Fri, Sep 18, 2015 at 2:11 PM, Eranga Udesh <eranga.erl@REDACTED>
> wrote:
>
>> Hi,
>>
>> I am experiencing the same issue in,
>>
>> - Erlang version : V6.4 , (Erlang/OTP 17 [erts-6.4] [source] [64-bit])
>> - OS version/architecture (32/64 bit) : Red Hat Enterprise Linux
>> Server release 6.6 (64 bit)
>>
>> The issue doesn't come always but after running the system for 3-10 days
>> can experience the tls_connection instances making high reductions and
>> consuming high memory as sent by Sanath. Since it's a production system, I
>> wasn't able to do much experiments, but applied the patches you sent, which
>> didn't solve this issue.
>>
>> There are 2 servers running Erlang VMs making SSL requests to the same
>> HTTPS host. When this issue happens, it happens in both the VMs. So it
>> could be triggered by some conditions in SSL or network connection. Even
>> though I left the VMs to run for a while to see if they can recover, but no
>> success. However if I restart the VMs, it starts to run as normal.
>>
>> So the summary is,
>>
>> - It triggers by some condition in remote SSL host or network
>> connection
>> - Erlang VMs don't recover itself
>> - Once restarted, it start working, which implies the issue is local.
>> i.e. in Erlang VM (tls_connection)
>>
>> We will try to recreate it and send you more details. I wonder if anybody
>> else is experiencing such issue.
>>
>> Tks,
>> - Eranga
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 18, 2015 at 1:36 PM, Ingela Andin <ingela.andin@REDACTED>
>> wrote:
>>
>>> Hi!
>>>
>>> As we are very busy with the release 18.1 I have not had time to try and
>>> recreate your problem. What version of OTP and the ssl application are you
>>> using?
>>> Can you reproduce the problem with the latest on github?
>>>
>>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>>
>>>
>>> On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>> wrote:
>>>
>>>> Hi Ingela,
>>>> Any update on this??
>>>> "Even apply your patch, *still problem is persist.* Any
>>>> more suggestions to solve this unexpected behavior ? "
>>>> Br,
>>>> Robert
>>>>
>>>> On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna <ahe.sanath@REDACTED
>>>> > wrote:
>>>>
>>>>> Hi Ingela,
>>>>> Even apply your patch, *still problem is persist.* Any
>>>>> more suggestions to solve this unexpected behavior ?
>>>>> Br,
>>>>> Robert
>>>>>
>>>>> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>>>> wrote:
>>>>>
>>>>>> Hi Ingela.
>>>>>> Tx a lot for your help & patch related to that.I'll inform you the
>>>>>> result after applying & testing patch.
>>>>>> Br,
>>>>>> Robert
>>>>>>
>>>>>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> It could be an bug that in the ssl application that I just fixed.
>>>>>>> The default session cache
>>>>>>> was violating the API, and this in turn made the mechanism for not
>>>>>>> registering a lot of equivalent
>>>>>>> sessions in the client fail.
>>>>>>>
>>>>>>> Here is the patch:
>>>>>>>
>>>>>>>
>>>>>>> diff --git a/lib/ssl/src/ssl_session.erl
>>>>>>> b/lib/ssl/src/ssl_session.erl
>>>>>>> index 1770faf..0d6cc93 100644
>>>>>>> --- a/lib/ssl/src/ssl_session.erl
>>>>>>> +++ b/lib/ssl/src/ssl_session.erl
>>>>>>> @@ -100,14 +100,14 @@ select_session([], _, _) ->
>>>>>>> no_session;
>>>>>>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert)
>>>>>>> ->
>>>>>>> IsNotResumable =
>>>>>>> - fun([_Id, Session]) ->
>>>>>>> + fun(Session) ->
>>>>>>> not (resumable(Session#session.is_resumable) andalso
>>>>>>> lists:member(Session#session.cipher_suite, Ciphers)
>>>>>>> andalso (OwnCert == Session#session.own_certificate))
>>>>>>> end,
>>>>>>> case lists:dropwhile(IsNotResumable, Sessions) of
>>>>>>> [] -> no_session;
>>>>>>> - [[Id, _]|_] -> Id
>>>>>>> + [Session | _] -> Session#session.session_id
>>>>>>> end.
>>>>>>>
>>>>>>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _,
>>>>>>> _) ->
>>>>>>> diff --git a/lib/ssl/src/ssl_session_cache.erl
>>>>>>> b/lib/ssl/src/ssl_session_cach
>>>>>>> e.erl
>>>>>>> index 11ed310..cfc48cd 100644
>>>>>>> --- a/lib/ssl/src/ssl_session_cache.erl
>>>>>>> +++ b/lib/ssl/src/ssl_session_cache.erl
>>>>>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) ->
>>>>>>>
>>>>>>> %%--------------------------------------------------------------------
>>>>>>> select_session(Cache, PartialKey) ->
>>>>>>> ets:select(Cache,
>>>>>>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]).
>>>>>>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]).
>>>>>>>
>>>>>>>
>>>>>>> %%--------------------------------------------------------------------
>>>>>>> %%% Internal functions
>>>>>>>
>>>>>>>
>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>>
>>>>>>>
>>>>>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna <ahe.sanath@REDACTED>:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I am running HTTP client using httpc Module to send both http and
>>>>>>>> https requests. normally sending arround 300 request per second without any
>>>>>>>> issue. however sometimes erlang node become very slow responsive. at that
>>>>>>>> time server load average is very high and using etop can identify
>>>>>>>> "tls_connection" process take more memory. when restart the erlang node its
>>>>>>>> become normal. as per my investigation normal time memory, processors,
>>>>>>>> loadAverage is not increasing. following is the HTTP request config
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync,
>>>>>>>> false}])
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> below is the etop output at that time
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> procs 1134 processes 1504844
>>>>>>>> code 9309
>>>>>>>>
>>>>>>>> runq 0 atom 420
>>>>>>>> ets 29692
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Pid Name or Initial Func Time Reds Memory MsgQ
>>>>>>>> Current Function
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0
>>>>>>>> gen_fsm:loop/7
>>>>>>>>
>>>>>>>>
>>>>>>>> can some one help me to solve this issue?
>>>>>>>>
>>>>>>>> Br,
>>>>>>>>
>>>>>>>> A.H.E. Robert
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> erlang-questions mailing list
>>>>>>>> erlang-questions@REDACTED
>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20151001/d3ed2b3b/attachment.htm>
More information about the erlang-questions
mailing list