[erlang-questions] Erlang OTP 18 memory leak / SSL

Sereysethy TOUCH touch.sereysethy@REDACTED
Mon Sep 21 16:33:07 CEST 2015


Dear all,

Finally I know what is the difference between Erlang 17 and Erlang 18 in
ssl module which caused a problem of memory. It is not really a memory leak
but simply the caching problem.

I take a look at the source code of ssl module (ssl_manager.erl) in OTP-17
& OTP-18. It is how they define the 24h in seconds. In OTP-17, somebody did
a wrong multiplication. While in OTP-18, the calculations were correct for
both 24h-in-sec & 24h-in-msec!

By the way, thanks to Ingela, Loïc and Jesper for helping me finding the
problems and solution.

Thanks,
T.

`Erlang OTP 17`

-define('24H_in_msec', 8640000).

-define('24H_in_sec', 8640).

-define(GEN_UNIQUE_ID_MAX_TRIES, 10).

-define(SESSION_VALIDATION_INTERVAL, 60000).

-define(CLEAR_PEM_CACHE, 120000).

-define(CLEAN_SESSION_DB, 60000).

-define(CLEAN_CERT_DB, 500).

-define(NOT_TO_BIG, 10).


`Erlang OTP 18`

-define('24H_in_msec', 86400000).

-define('24H_in_sec', 86400).

-define(GEN_UNIQUE_ID_MAX_TRIES, 10).

-define(SESSION_VALIDATION_INTERVAL, 60000).

-define(CLEAR_PEM_CACHE, 120000).

-define(CLEAN_SESSION_DB, 60000).

-define(CLEAN_CERT_DB, 500).

-define(NOT_TO_BIG, 10).

On Mon, Sep 21, 2015 at 6:00 PM, Sereysethy TOUCH <
touch.sereysethy@REDACTED> wrote:

> I got it LoÏc.
>
> Thanks.
>
> On Mon, Sep 21, 2015 at 5:52 PM, Sereysethy TOUCH <
> touch.sereysethy@REDACTED> wrote:
>
>> Something like this?
>>
>> {release, {ws_user, "1"},
>>
>>         [ws_user]}.
>>
>> {sys_config, "./config/sys.config"}.
>>
>> %{paths, ["../deps/"]}.
>>
>> % generate startup script, yes
>>
>> {extended_start_script, true}.
>>
>> On Mon, Sep 21, 2015 at 5:46 PM, Loïc Hoguin <essen@REDACTED> wrote:
>>
>>> Add {sys_config, "path/to/sys.config"} to your relx.config, then create
>>> that sys.config file with the right options and run make.
>>>
>>> On 09/21/2015 12:44 PM, Sereysethy TOUCH wrote:
>>>
>>>> Hello,
>>>>
>>>> You meant _rel or I need to create a folder "rel" and put sys.config
>>>> inside?
>>>>
>>>> Sethy
>>>>
>>>> On Mon, Sep 21, 2015 at 5:09 PM, Loïc Hoguin <essen@REDACTED
>>>> <mailto:essen@REDACTED>> wrote:
>>>>
>>>>     On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote:
>>>>
>>>>         Hello,
>>>>
>>>>         My application is both TLS-client & server. It is a web server
>>>> which
>>>>         allows client browser to connect to server using websocket. I
>>>>         use cowboy
>>>>         as web server & web socket server.
>>>>
>>>>         In erlang 17, our application runs for weeks and there is no
>>>> memory
>>>>         issue. We only restart it whenever we have a new update. But
>>>>         since we
>>>>         update to elang 18, we need to restart our application every 2
>>>>         hours.
>>>>
>>>>         I use erlang.mk <http://erlang.mk> <http://erlang.mk> to
>>>> compile
>>>>         our application, I am not
>>>>         sure how to set application env. settings. Is it the same if I
>>>> use
>>>>         erlang.mk <http://erlang.mk> <http://erlang.mk>?
>>>>
>>>>
>>>>     If you run a release, you can put the configuration in the
>>>>     sys.config file (by default rel/sys.config) and run make again. See
>>>>     http://www.erlang.org/doc/man/config.html
>>>>
>>>>
>>>>
>>>>         Thanks,
>>>>         Sethy
>>>>
>>>>         On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin
>>>>         <ingela.andin@REDACTED <mailto:ingela.andin@REDACTED>
>>>>         <mailto:ingela.andin@REDACTED <mailto:ingela.andin@REDACTED
>>>> >>>
>>>>         wrote:
>>>>
>>>>              Hi!
>>>>
>>>>
>>>>              2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH
>>>>              <touch.sereysethy@REDACTED
>>>>         <mailto:touch.sereysethy@REDACTED>
>>>>         <mailto:touch.sereysethy@REDACTED
>>>>
>>>>         <mailto:touch.sereysethy@REDACTED>>>:
>>>>
>>>>
>>>>                  Hello,
>>>>
>>>>                  One question, why did my application work fine on
>>>>         Erlang 17.5.x
>>>>                  branch?
>>>>
>>>>
>>>>              Well I am not sure what your application does.
>>>>
>>>>              If you have a client application that spawns a lot of
>>>> parallel
>>>>              connections to the same host, not waiting for one to
>>>>         succeed before
>>>>              spawning,
>>>>              the client session table could grow a lot. However I have
>>>>              implemented a feature to only save unique TLS-sessions in
>>>> this
>>>>              senario to avoid this, and there was a bug so that this
>>>>         feature did
>>>>              not work, that I have fix in the maint branch. This bug
>>>>         however was
>>>>              in 17 as well as 18.
>>>>
>>>>              However the ets info you sent suggest you have a server
>>>>         application
>>>>              and then setting session_liftime can help keeping the
>>>> servers
>>>>              session table
>>>>              smaller. This also is true for clients, that can get a big
>>>>         table
>>>>              also through connecting to many diffrent servers.
>>>>
>>>>              Maybe your application can be both a TLS-client and server?
>>>>         There is
>>>>              no obvious reason from what you told us so far that it
>>>>         should work
>>>>              better in 17 then 18.
>>>>
>>>>
>>>>                  If I want to set this session_lifetime option, where
>>>>         should put it?
>>>>
>>>>
>>>>              It is an application environment setting. Search for app in
>>>>         the docs.
>>>>
>>>>
>>>>                  So there is no use that I should build the erlang 18
>>>>         from maint
>>>>                  branch?
>>>>
>>>>
>>>>
>>>>              It is always a good thing to try the latest version, and
>>>> if you
>>>>              still have a problem it would be good if you could try
>>>>         providing a
>>>>              way to reproduce the problem.
>>>>
>>>>              Regards Ingela Erlang/OTP tema Ericsson AB
>>>>
>>>>
>>>>                  Thanks,
>>>>                  Sethy
>>>>
>>>>                  On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin
>>>>                  <ingela.andin@REDACTED <mailto:ingela.andin@REDACTED
>>>> >
>>>>         <mailto:ingela.andin@REDACTED <mailto:ingela.andin@REDACTED
>>>> >>>
>>>>         wrote:
>>>>
>>>>                      Hello,
>>>>
>>>>                      2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH
>>>>                      <touch.sereysethy@REDACTED
>>>>         <mailto:touch.sereysethy@REDACTED>
>>>>                      <mailto:touch.sereysethy@REDACTED
>>>>
>>>>         <mailto:touch.sereysethy@REDACTED>>>:
>>>>
>>>>
>>>>                          Hello,
>>>>
>>>>                          I did as what you told me by getting a shell
>>>>         console on
>>>>                          the node.
>>>>
>>>>                          I run memory(). and see ets ate the most
>>>>         memory. And it
>>>>                          increased overtime. I run ets:i(). and I found
>>>>         out the
>>>>                          ssl_manager took a big chunk of what memory
>>>> used.
>>>>
>>>>                          32794           client_ssl_otp_session_cache
>>>>         ordered_set
>>>>                          0      89       ssl_manager
>>>>                          36891           server_ssl_otp_session_cache
>>>>         ordered_set
>>>>                          1564   67995    ssl_manager
>>>>                          40993
>>>>           httpc_manager__session_cookie_db bag   0
>>>>                                299      httpc_manager
>>>>
>>>>                          So I think there is a problem with ssl_manager
>>>>         in Erlang
>>>>                          OTP 18.
>>>>
>>>>                          Any workarounds?
>>>>
>>>>
>>>>
>>>>                      Ok, the bug I fixed with the session table is on
>>>>         the client
>>>>                      side.  But your big ets table is on the server
>>>> side.
>>>>                      Session data is by default  fairly long lived (24 h
>>>>         that is
>>>>                      the max recommended time to keep a session by the
>>>> RFC),
>>>>                      you can set the ssl application variable
>>>>         session_lifetime to
>>>>                      make  its lifetime shorter which should mitigate
>>>>         your problem.
>>>>                      We have a backlog item to make a configurable
>>>>         threshold of
>>>>                      max sessions allowed to be stored in the table,
>>>>         which I am
>>>>                      sure will be
>>>>                      implemented fairly soon but alas  not for 18.1.
>>>>
>>>>                      Regards Ingela Erlang/OTP Team - Ericsson AB
>>>>
>>>>                          Sethy
>>>>
>>>>                          On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis
>>>>         Andersen
>>>>                          <jesper.louis.andersen@REDACTED
>>>>         <mailto:jesper.louis.andersen@REDACTED>
>>>>                          <mailto:jesper.louis.andersen@REDACTED
>>>>         <mailto:jesper.louis.andersen@REDACTED>>> wrote:
>>>>
>>>>
>>>>                              On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy
>>>>         TOUCH
>>>>                              <touch.sereysethy@REDACTED
>>>>         <mailto:touch.sereysethy@REDACTED>
>>>>                              <mailto:touch.sereysethy@REDACTED
>>>>
>>>>         <mailto:touch.sereysethy@REDACTED>>> wrote:
>>>>
>>>>                                  I just recently updated Erlang to
>>>>         latest version
>>>>                                  OTP 18 on Ubuntu server. It uses cowboy
>>>>                                  (websocket), ranch, ssl, erlydtl &
>>>>         rabbitmq. It
>>>>                                  used to work fine in OTP 17. The
>>>> program is
>>>>                                  correctly compiled but during the
>>>>         execution the
>>>>                                  memory kept increasing. I need to
>>>>         restart the
>>>>                                  process every one or two hours to free
>>>>         some memory.
>>>>
>>>>
>>>>                              1. Get a shell console on the node
>>>>                              2. call 'memory().' in the shell
>>>>                              3. If it reports ets as the winner, call
>>>>                              4. ets:i().
>>>>
>>>>                              This will give you a nice overview for
>>>>         where to look
>>>>                              and verify it is the SSL problem.
>>>>
>>>>
>>>>                              --
>>>>                              J.
>>>>
>>>>
>>>>
>>>>                          _______________________________________________
>>>>                          erlang-questions mailing list
>>>>         erlang-questions@REDACTED <mailto:erlang-questions@REDACTED
>>>> >
>>>>                          <mailto:erlang-questions@REDACTED
>>>>         <mailto:erlang-questions@REDACTED>>
>>>>         http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>         _______________________________________________
>>>>         erlang-questions mailing list
>>>>         erlang-questions@REDACTED <mailto:erlang-questions@REDACTED
>>>> >
>>>>         http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>>
>>>>     --
>>>>     Loïc Hoguin
>>>>     http://ninenines.eu
>>>>     Author of The Erlanger Playbook,
>>>>     A book about software development using Erlang
>>>>
>>>>
>>>>
>>> --
>>> Loïc Hoguin
>>> http://ninenines.eu
>>> Author of The Erlanger Playbook,
>>> A book about software development using Erlang
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150921/c8aec892/attachment.htm>


More information about the erlang-questions mailing list