[erlang-questions] Fwd: Unexpected behavior of HTTPC module - tls_connection process take more memory
Eranga Udesh
eranga.erl@REDACTED
Fri Sep 18 10:41:20 CEST 2015
Hi,
I am experiencing the same issue in,
- Erlang version : V6.4 , (Erlang/OTP 17 [erts-6.4] [source] [64-bit])
- OS version/architecture (32/64 bit) : Red Hat Enterprise Linux Server
release 6.6 (64 bit)
The issue doesn't come always but after running the system for 3-10 days
can experience the tls_connection instances making high reductions and
consuming high memory as sent by Sanath. Since it's a production system, I
wasn't able to do much experiments, but applied the patches you sent, which
didn't solve this issue.
There are 2 servers running Erlang VMs making SSL requests to the same
HTTPS host. When this issue happens, it happens in both the VMs. So it
could be triggered by some conditions in SSL or network connection. Even
though I left the VMs to run for a while to see if they can recover, but no
success. However if I restart the VMs, it starts to run as normal.
So the summary is,
- It triggers by some condition in remote SSL host or network connection
- Erlang VMs don't recover itself
- Once restarted, it start working, which implies the issue is local.
i.e. in Erlang VM (tls_connection)
We will try to recreate it and send you more details. I wonder if anybody
else is experiencing such issue.
Tks,
- Eranga
On Fri, Sep 18, 2015 at 1:36 PM, Ingela Andin <ingela.andin@REDACTED>
wrote:
> Hi!
>
> As we are very busy with the release 18.1 I have not had time to try and
> recreate your problem. What version of OTP and the ssl application are you
> using?
> Can you reproduce the problem with the latest on github?
>
> Regards Ingela Erlang/OTP Team - Ericsson AB
>
>
> On Thu, Sep 17, 2015 at 8:05 PM, Sanath Prasanna <ahe.sanath@REDACTED>
> wrote:
>
>> Hi Ingela,
>> Any update on this??
>> "Even apply your patch, *still problem is persist.* Any
>> more suggestions to solve this unexpected behavior ? "
>> Br,
>> Robert
>>
>> On Thu, Sep 17, 2015 at 12:39 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>> wrote:
>>
>>> Hi Ingela,
>>> Even apply your patch, *still problem is persist.* Any
>>> more suggestions to solve this unexpected behavior ?
>>> Br,
>>> Robert
>>>
>>> On Tue, Sep 8, 2015 at 1:52 PM, Sanath Prasanna <ahe.sanath@REDACTED>
>>> wrote:
>>>
>>>> Hi Ingela.
>>>> Tx a lot for your help & patch related to that.I'll inform you the
>>>> result after applying & testing patch.
>>>> Br,
>>>> Robert
>>>>
>>>> On Tue, Sep 8, 2015 at 1:31 PM, Ingela Andin <ingela.andin@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> It could be an bug that in the ssl application that I just fixed. The
>>>>> default session cache
>>>>> was violating the API, and this in turn made the mechanism for not
>>>>> registering a lot of equivalent
>>>>> sessions in the client fail.
>>>>>
>>>>> Here is the patch:
>>>>>
>>>>>
>>>>> diff --git a/lib/ssl/src/ssl_session.erl b/lib/ssl/src/ssl_session.erl
>>>>> index 1770faf..0d6cc93 100644
>>>>> --- a/lib/ssl/src/ssl_session.erl
>>>>> +++ b/lib/ssl/src/ssl_session.erl
>>>>> @@ -100,14 +100,14 @@ select_session([], _, _) ->
>>>>> no_session;
>>>>> select_session(Sessions, #ssl_options{ciphers = Ciphers}, OwnCert) ->
>>>>> IsNotResumable =
>>>>> - fun([_Id, Session]) ->
>>>>> + fun(Session) ->
>>>>> not (resumable(Session#session.is_resumable) andalso
>>>>> lists:member(Session#session.cipher_suite, Ciphers)
>>>>> andalso (OwnCert == Session#session.own_certificate))
>>>>> end,
>>>>> case lists:dropwhile(IsNotResumable, Sessions) of
>>>>> [] -> no_session;
>>>>> - [[Id, _]|_] -> Id
>>>>> + [Session | _] -> Session#session.session_id
>>>>> end.
>>>>>
>>>>> is_resumable(_, _, #ssl_options{reuse_sessions = false}, _, _, _, _)
>>>>> ->
>>>>> diff --git a/lib/ssl/src/ssl_session_cache.erl
>>>>> b/lib/ssl/src/ssl_session_cach
>>>>> e.erl
>>>>> index 11ed310..cfc48cd 100644
>>>>> --- a/lib/ssl/src/ssl_session_cache.erl
>>>>> +++ b/lib/ssl/src/ssl_session_cache.erl
>>>>> @@ -83,7 +83,7 @@ foldl(Fun, Acc0, Cache) ->
>>>>> %%--------------------------------------------------------------------
>>>>> select_session(Cache, PartialKey) ->
>>>>> ets:select(Cache,
>>>>> - [{{{PartialKey,'$1'}, '$2'},[],['$$']}]).
>>>>> + [{{{PartialKey,'_'}, '$1'},[],['$1']}]).
>>>>>
>>>>> %%--------------------------------------------------------------------
>>>>> %%% Internal functions
>>>>>
>>>>>
>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>
>>>>>
>>>>> 2015-09-07 5:52 GMT+02:00 Sanath Prasanna <ahe.sanath@REDACTED>:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I am running HTTP client using httpc Module to send both http and
>>>>>> https requests. normally sending arround 300 request per second without any
>>>>>> issue. however sometimes erlang node become very slow responsive. at that
>>>>>> time server load average is very high and using etop can identify
>>>>>> "tls_connection" process take more memory. when restart the erlang node its
>>>>>> become normal. as per my investigation normal time memory, processors,
>>>>>> loadAverage is not increasing. following is the HTTP request config
>>>>>>
>>>>>>
>>>>>>
>>>>>> httpc:request(Method, Request, [{timeout, TimeoutTime}], [{sync,
>>>>>> false}])
>>>>>>
>>>>>>
>>>>>>
>>>>>> below is the etop output at that time
>>>>>>
>>>>>>
>>>>>>
>>>>>> procs 1134 processes 1504844
>>>>>> code 9309
>>>>>>
>>>>>> runq 0 atom 420
>>>>>> ets 29692
>>>>>>
>>>>>>
>>>>>>
>>>>>> Pid Name or Initial Func Time Reds Memory MsgQ
>>>>>> Current Function
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------
>>>>>>
>>>>>> <5490.26428.14>tls_connection:init/ '-' 733224580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26429.14>tls_connection:init/ '-' 1328524580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26430.14>tls_connection:init/ '-' 528924580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26431.14>tls_connection:init/ '-' 1432224580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26432.14>tls_connection:init/ '-' 024580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26433.14>tls_connection:init/ '-' 024580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26434.14>tls_connection:init/ '-' 024580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26435.14>tls_connection:init/ '-' 024580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>> <5490.26436.14>tls_connection:init/ '-' 024580768 0
>>>>>> gen_fsm:loop/7
>>>>>>
>>>>>>
>>>>>> can some one help me to solve this issue?
>>>>>>
>>>>>> Br,
>>>>>>
>>>>>> A.H.E. Robert
>>>>>>
>>>>>> _______________________________________________
>>>>>> erlang-questions mailing list
>>>>>> erlang-questions@REDACTED
>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150918/104bb10a/attachment.htm>
More information about the erlang-questions
mailing list