[erlang-questions] Erlang OTP 18 memory leak / SSL
Sereysethy TOUCH
touch.sereysethy@REDACTED
Mon Sep 21 16:33:07 CEST 2015
Dear all,
Finally I know what is the difference between Erlang 17 and Erlang 18 in
ssl module which caused a problem of memory. It is not really a memory leak
but simply the caching problem.
I take a look at the source code of ssl module (ssl_manager.erl) in OTP-17
& OTP-18. It is how they define the 24h in seconds. In OTP-17, somebody did
a wrong multiplication. While in OTP-18, the calculations were correct for
both 24h-in-sec & 24h-in-msec!
By the way, thanks to Ingela, Loïc and Jesper for helping me finding the
problems and solution.
Thanks,
T.
`Erlang OTP 17`
-define('24H_in_msec', 8640000).
-define('24H_in_sec', 8640).
-define(GEN_UNIQUE_ID_MAX_TRIES, 10).
-define(SESSION_VALIDATION_INTERVAL, 60000).
-define(CLEAR_PEM_CACHE, 120000).
-define(CLEAN_SESSION_DB, 60000).
-define(CLEAN_CERT_DB, 500).
-define(NOT_TO_BIG, 10).
`Erlang OTP 18`
-define('24H_in_msec', 86400000).
-define('24H_in_sec', 86400).
-define(GEN_UNIQUE_ID_MAX_TRIES, 10).
-define(SESSION_VALIDATION_INTERVAL, 60000).
-define(CLEAR_PEM_CACHE, 120000).
-define(CLEAN_SESSION_DB, 60000).
-define(CLEAN_CERT_DB, 500).
-define(NOT_TO_BIG, 10).
On Mon, Sep 21, 2015 at 6:00 PM, Sereysethy TOUCH <
touch.sereysethy@REDACTED> wrote:
> I got it LoÏc.
>
> Thanks.
>
> On Mon, Sep 21, 2015 at 5:52 PM, Sereysethy TOUCH <
> touch.sereysethy@REDACTED> wrote:
>
>> Something like this?
>>
>> {release, {ws_user, "1"},
>>
>> [ws_user]}.
>>
>> {sys_config, "./config/sys.config"}.
>>
>> %{paths, ["../deps/"]}.
>>
>> % generate startup script, yes
>>
>> {extended_start_script, true}.
>>
>> On Mon, Sep 21, 2015 at 5:46 PM, Loïc Hoguin <essen@REDACTED> wrote:
>>
>>> Add {sys_config, "path/to/sys.config"} to your relx.config, then create
>>> that sys.config file with the right options and run make.
>>>
>>> On 09/21/2015 12:44 PM, Sereysethy TOUCH wrote:
>>>
>>>> Hello,
>>>>
>>>> You meant _rel or I need to create a folder "rel" and put sys.config
>>>> inside?
>>>>
>>>> Sethy
>>>>
>>>> On Mon, Sep 21, 2015 at 5:09 PM, Loïc Hoguin <essen@REDACTED
>>>> <mailto:essen@REDACTED>> wrote:
>>>>
>>>> On 09/21/2015 12:05 PM, Sereysethy TOUCH wrote:
>>>>
>>>> Hello,
>>>>
>>>> My application is both TLS-client & server. It is a web server
>>>> which
>>>> allows client browser to connect to server using websocket. I
>>>> use cowboy
>>>> as web server & web socket server.
>>>>
>>>> In erlang 17, our application runs for weeks and there is no
>>>> memory
>>>> issue. We only restart it whenever we have a new update. But
>>>> since we
>>>> update to elang 18, we need to restart our application every 2
>>>> hours.
>>>>
>>>> I use erlang.mk <http://erlang.mk> <http://erlang.mk> to
>>>> compile
>>>> our application, I am not
>>>> sure how to set application env. settings. Is it the same if I
>>>> use
>>>> erlang.mk <http://erlang.mk> <http://erlang.mk>?
>>>>
>>>>
>>>> If you run a release, you can put the configuration in the
>>>> sys.config file (by default rel/sys.config) and run make again. See
>>>> http://www.erlang.org/doc/man/config.html
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Sethy
>>>>
>>>> On Mon, Sep 21, 2015 at 4:54 PM, Ingela Andin
>>>> <ingela.andin@REDACTED <mailto:ingela.andin@REDACTED>
>>>> <mailto:ingela.andin@REDACTED <mailto:ingela.andin@REDACTED
>>>> >>>
>>>> wrote:
>>>>
>>>> Hi!
>>>>
>>>>
>>>> 2015-09-21 10:09 GMT+02:00 Sereysethy TOUCH
>>>> <touch.sereysethy@REDACTED
>>>> <mailto:touch.sereysethy@REDACTED>
>>>> <mailto:touch.sereysethy@REDACTED
>>>>
>>>> <mailto:touch.sereysethy@REDACTED>>>:
>>>>
>>>>
>>>> Hello,
>>>>
>>>> One question, why did my application work fine on
>>>> Erlang 17.5.x
>>>> branch?
>>>>
>>>>
>>>> Well I am not sure what your application does.
>>>>
>>>> If you have a client application that spawns a lot of
>>>> parallel
>>>> connections to the same host, not waiting for one to
>>>> succeed before
>>>> spawning,
>>>> the client session table could grow a lot. However I have
>>>> implemented a feature to only save unique TLS-sessions in
>>>> this
>>>> senario to avoid this, and there was a bug so that this
>>>> feature did
>>>> not work, that I have fix in the maint branch. This bug
>>>> however was
>>>> in 17 as well as 18.
>>>>
>>>> However the ets info you sent suggest you have a server
>>>> application
>>>> and then setting session_liftime can help keeping the
>>>> servers
>>>> session table
>>>> smaller. This also is true for clients, that can get a big
>>>> table
>>>> also through connecting to many diffrent servers.
>>>>
>>>> Maybe your application can be both a TLS-client and server?
>>>> There is
>>>> no obvious reason from what you told us so far that it
>>>> should work
>>>> better in 17 then 18.
>>>>
>>>>
>>>> If I want to set this session_lifetime option, where
>>>> should put it?
>>>>
>>>>
>>>> It is an application environment setting. Search for app in
>>>> the docs.
>>>>
>>>>
>>>> So there is no use that I should build the erlang 18
>>>> from maint
>>>> branch?
>>>>
>>>>
>>>>
>>>> It is always a good thing to try the latest version, and
>>>> if you
>>>> still have a problem it would be good if you could try
>>>> providing a
>>>> way to reproduce the problem.
>>>>
>>>> Regards Ingela Erlang/OTP tema Ericsson AB
>>>>
>>>>
>>>> Thanks,
>>>> Sethy
>>>>
>>>> On Mon, Sep 21, 2015 at 2:51 PM, Ingela Andin
>>>> <ingela.andin@REDACTED <mailto:ingela.andin@REDACTED
>>>> >
>>>> <mailto:ingela.andin@REDACTED <mailto:ingela.andin@REDACTED
>>>> >>>
>>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> 2015-09-20 15:24 GMT+02:00 Sereysethy TOUCH
>>>> <touch.sereysethy@REDACTED
>>>> <mailto:touch.sereysethy@REDACTED>
>>>> <mailto:touch.sereysethy@REDACTED
>>>>
>>>> <mailto:touch.sereysethy@REDACTED>>>:
>>>>
>>>>
>>>> Hello,
>>>>
>>>> I did as what you told me by getting a shell
>>>> console on
>>>> the node.
>>>>
>>>> I run memory(). and see ets ate the most
>>>> memory. And it
>>>> increased overtime. I run ets:i(). and I found
>>>> out the
>>>> ssl_manager took a big chunk of what memory
>>>> used.
>>>>
>>>> 32794 client_ssl_otp_session_cache
>>>> ordered_set
>>>> 0 89 ssl_manager
>>>> 36891 server_ssl_otp_session_cache
>>>> ordered_set
>>>> 1564 67995 ssl_manager
>>>> 40993
>>>> httpc_manager__session_cookie_db bag 0
>>>> 299 httpc_manager
>>>>
>>>> So I think there is a problem with ssl_manager
>>>> in Erlang
>>>> OTP 18.
>>>>
>>>> Any workarounds?
>>>>
>>>>
>>>>
>>>> Ok, the bug I fixed with the session table is on
>>>> the client
>>>> side. But your big ets table is on the server
>>>> side.
>>>> Session data is by default fairly long lived (24 h
>>>> that is
>>>> the max recommended time to keep a session by the
>>>> RFC),
>>>> you can set the ssl application variable
>>>> session_lifetime to
>>>> make its lifetime shorter which should mitigate
>>>> your problem.
>>>> We have a backlog item to make a configurable
>>>> threshold of
>>>> max sessions allowed to be stored in the table,
>>>> which I am
>>>> sure will be
>>>> implemented fairly soon but alas not for 18.1.
>>>>
>>>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>>>
>>>> Sethy
>>>>
>>>> On Sun, Sep 20, 2015 at 2:22 AM, Jesper Louis
>>>> Andersen
>>>> <jesper.louis.andersen@REDACTED
>>>> <mailto:jesper.louis.andersen@REDACTED>
>>>> <mailto:jesper.louis.andersen@REDACTED
>>>> <mailto:jesper.louis.andersen@REDACTED>>> wrote:
>>>>
>>>>
>>>> On Sat, Sep 19, 2015 at 6:34 PM, Sereysethy
>>>> TOUCH
>>>> <touch.sereysethy@REDACTED
>>>> <mailto:touch.sereysethy@REDACTED>
>>>> <mailto:touch.sereysethy@REDACTED
>>>>
>>>> <mailto:touch.sereysethy@REDACTED>>> wrote:
>>>>
>>>> I just recently updated Erlang to
>>>> latest version
>>>> OTP 18 on Ubuntu server. It uses cowboy
>>>> (websocket), ranch, ssl, erlydtl &
>>>> rabbitmq. It
>>>> used to work fine in OTP 17. The
>>>> program is
>>>> correctly compiled but during the
>>>> execution the
>>>> memory kept increasing. I need to
>>>> restart the
>>>> process every one or two hours to free
>>>> some memory.
>>>>
>>>>
>>>> 1. Get a shell console on the node
>>>> 2. call 'memory().' in the shell
>>>> 3. If it reports ets as the winner, call
>>>> 4. ets:i().
>>>>
>>>> This will give you a nice overview for
>>>> where to look
>>>> and verify it is the SSL problem.
>>>>
>>>>
>>>> --
>>>> J.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED
>>>> >
>>>> <mailto:erlang-questions@REDACTED
>>>> <mailto:erlang-questions@REDACTED>>
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED
>>>> >
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>>
>>>> --
>>>> Loïc Hoguin
>>>> http://ninenines.eu
>>>> Author of The Erlanger Playbook,
>>>> A book about software development using Erlang
>>>>
>>>>
>>>>
>>> --
>>> Loïc Hoguin
>>> http://ninenines.eu
>>> Author of The Erlanger Playbook,
>>> A book about software development using Erlang
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150921/c8aec892/attachment.htm>
More information about the erlang-questions
mailing list