<div dir="ltr">Hi,<div><br></div><div>I used to face this kind of problem when I updated erlang from OTP 17.x.x to 18 but I think the problem was fixed already in 18.2</div><div><br></div><div>Here is what I found out:</div><div><br></div><div>My problem was that `ets` table keeps growing and it is caused by `server_ssl_otp_session_cache`</div>
<p class=""><span class="">I traced it back to the source code of Erlang 17 & Erlang 18 release branch, in ssl module/ssl_manager.erl. There was a mistake in calculation of 24H_in_msec in Erlang 17.</span></p><p class=""><span class=""><br></span></p><p class=""><span class="">`Erlang OTP 17`</span></p>
<p class=""><span class=""><br></span></p><p class=""><span class="">-define('24H_in_msec', <font color="#ff0000">8640000</font>).</span></p>
<p class=""><span class="">-define('24H_in_sec', 8640).</span></p>
<p class=""><span class="">-define(GEN_UNIQUE_ID_MAX_TRIES, 10).</span></p>
<p class=""><span class="">-define(SESSION_VALIDATION_INTERVAL, 60000).</span></p>
<p class=""><span class="">-define(CLEAR_PEM_CACHE, 120000).</span></p>
<p class=""><span class="">-define(CLEAN_SESSION_DB, 60000).</span></p>
<p class=""><span class="">-define(CLEAN_CERT_DB, 500).</span></p>
<p class=""><span class="">-define(NOT_TO_BIG, 10).</span></p>
<p class=""><span class=""></span><br></p>
<p class=""><span class="">`Erlang OTP 18`</span></p>
<p class=""><span class=""></span><br></p>
<p class=""><span class="">-define('24H_in_msec', <font color="#ff0000">86400000</font>).</span></p>
<p class=""><span class="">-define('24H_in_sec', 86400).</span></p>
<p class=""><span class="">-define(GEN_UNIQUE_ID_MAX_TRIES, 10).</span></p>
<p class=""><span class="">-define(SESSION_VALIDATION_INTERVAL, 60000).</span></p>
<p class=""><span class="">-define(CLEAR_PEM_CACHE, 120000).</span></p>
<p class=""><span class="">-define(CLEAN_SESSION_DB, 60000).</span></p>
<p class=""><span class="">-define(CLEAN_CERT_DB, 500).</span></p>
<p class=""><span class="">-define(NOT_TO_BIG, 10).</span></p><p class=""><span class=""><br></span></p><p class=""><span class="">So you can try to pass </span> `session_lifetime` to 30mn (1800 seconds) to `ssl` application (module).</p><p class="">Sethy</p>
</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 28, 2016 at 5:36 PM, <span dir="ltr"><<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
Could you please clarify one moment. Is the ssl session cache used only<br>
when reuse_sessions is set to true? Is there a way to turn off the cache<br>
completely? I'm trying to figure out what settings are needed for<br>
thousands of short-lived connections.<br>
<br>
28/04/16 12:21, Ingela Andin пишет:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi!<br>
<br>
<br>
2016-04-27 15:44 GMT+02:00 <<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a> <mailto:<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a>>>:<span class=""><br>
<br>
Hello,<br>
<br>
The reason was session_cache_server_max (defaults to 1000).<br>
The ssl_manager quickly get filled by<br>
<br>
{'$gen_cast',{invalidate_session,8443,{session,<<...>>,...}}}<br>
{delayed_clean_session,{8443,<<14,50,229,...>>},40986}<br>
<br>
messages.<br>
<br>
I see, we will looking into mitigating this effect.<br>
<br>
Regards Ingela Erlang/OTP team - Ericsson AB<br>
<br>
<br>
<br>
27/04/16 13:29, Ingela Andin пишет:<br>
<br>
Hi!<br>
<br>
You could try using the observer application to find out more<br>
information about your system state.<br>
There is no apparent reason why ssl-7.3 should consume a lot<br>
more memory<br>
than ssl-6.0.1.2.<br>
But a lot has append between the versions and heavy load may<br>
find corner<br>
cases that where missed.<br>
<br>
Regards Ingela Erlang/OTP Team - Ericsson AB<br>
<br>
<br>
2016-04-25 14:35 GMT+02:00 <<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a><br></span>
<mailto:<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a>> <mailto:<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a><div><div class="h5"><br>
<mailto:<a href="mailto:vicbaz@yandex.ru" target="_blank">vicbaz@yandex.ru</a>>>>:<br>
<br>
<br>
Hello,<br>
<br>
When I run<br>
<a href="https://github.com/ninenines/cowboy/tree/1.0.4/examples/ssl_hello_world" rel="noreferrer" target="_blank">https://github.com/ninenines/cowboy/tree/1.0.4/examples/ssl_hello_world</a><br>
under load (using <a href="https://github.com/JoeDog/siege" rel="noreferrer" target="_blank">https://github.com/JoeDog/siege</a>) I found<br>
a big<br>
difference in memory consumption between OTP-17.5 and<br>
OTP-18.3. I know<br>
the ssl application has many changes in new versions. Is<br>
this memory<br>
usage normal now or can be reduced using some ssl<br>
application settings?<br>
<br>
I use OTP-17.5.6.9 and OTP-18.3.1.<br>
<br>
<br>
================================================================================<br>
Erlang/OTP 17 [erts-6.4.1.6] [source] [64-bit] [smp:2:2]<br>
[async-threads:10] [hipe] [kernel-poll:false]<br>
<br>
1> application:which_applications().<br>
[{ssl_hello_world,"Cowboy Hello World example with SSL.",<br>
"1"},<br>
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},<br>
{ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},<br>
{cowlib,"Support library for manipulating Web protocols.",<br>
"1.0.2"},<br>
{ssl,"Erlang/OTP SSL application","6.0.1.2"},<br>
{public_key,"Public key infrastructure","0.23"},<br>
{crypto,"CRYPTO","3.5"},<br>
{asn1,"The Erlang ASN1 compiler version 3.0.4","3.0.4"},<br>
{stdlib,"ERTS CXC 138 10","2.4"},<br>
{kernel,"ERTS CXC 138 10","3.2.0.1"}]<br>
<br>
2> erlang:memory().<br>
[{total,43913840},<br>
{processes,8036312},<br>
{processes_used,8030744},<br>
{system,35877528},<br>
{atom,470537},<br>
{atom_used,455904},<br>
{binary,1498800},<br>
{code,11074476},<br>
{ets,14516424}]<br>
<br>
3> recon_alloc:memory(used).<br>
45519464<br>
<br>
4> recon_alloc:memory(usage).<br>
0.7212344918526264<br>
<br>
5> recon:proc_window(memory, 10, 500).<br>
[{<0.25910.2>,68120,<br>
[{current_function,{gen_fsm,loop,7}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.25908.2>,68120,<br>
[{current_function,{gen_fsm,loop,7}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.284.0>,15808,<br>
[ssl_manager,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.285.0>,12712,<br>
[tls_connection_sup,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.25909.2>,5888,<br>
[{current_function,{gen,do_call,4}},<br>
{initial_call,{cowboy_protocol,init,4}}]},<br>
{<0.324.0>,4888,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.302.0>,4808,<br>
[{current_function,{ranch_conns_sup,loop,4}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.330.0>,3016,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.25911.2>,2848,<br>
[{current_function,{gen,do_call,4}},<br>
{initial_call,{cowboy_protocol,init,4}}]},<br>
{<0.340.0>,1872,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]}]<br>
<br>
6> recon:proc_window(reductions, 10, 500).<br>
[{<0.2634.3>,11161,<br>
[{current_function,{gen_fsm,loop,7}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.2633.3>,9747,<br>
[{current_function,{crypto,int_to_bin_pos,2}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.284.0>,4097,<br>
[ssl_manager,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.285.0>,3314,<br>
[tls_connection_sup,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.302.0>,1003,<br>
[{current_function,{ranch_conns_sup,loop,4}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.378.0>,426,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.372.0>,425,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.379.0>,425,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.377.0>,425,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.343.0>,425,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]}]<br>
<br>
RES in htop ~50-60M<br>
<br>
<br>
================================================================================<br>
<br>
Erlang/OTP 18 [erts-7.3.1] [source] [64-bit] [smp:2:2]<br>
[async-threads:10] [hipe] [kernel-poll:false]<br>
<br>
1> application:which_applications().<br>
[{ssl_hello_world,"Cowboy Hello World example with SSL.",<br>
"1"},<br>
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},<br>
{ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},<br>
{cowlib,"Support library for manipulating Web protocols.",<br>
"1.0.2"},<br>
{ssl,"Erlang/OTP SSL application","7.3"},<br>
{public_key,"Public key infrastructure","1.1.1"},<br>
{crypto,"CRYPTO","3.6.3"},<br>
{asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"},<br>
{recon,"Diagnostic tools for production use","2.3.1"},<br>
{stdlib,"ERTS CXC 138 10","2.8"},<br>
{kernel,"ERTS CXC 138 10","4.2"}]<br>
<br>
2> erlang:memory().<br>
[{total,949153560},<br>
{processes,902969888},<br>
{processes_used,902558312},<br>
{system,46183672},<br>
{atom,437761},<br>
{atom_used,432875},<br>
{binary,48320},<br>
{code,11407283},<br>
{ets,883640}]<br>
<br>
3> recon_alloc:memory(used).<br>
863194560<br>
<br>
4> recon_alloc:memory(usage).<br>
0.8005009562139471<br>
<br>
5> recon:proc_window(memory, 10, 500).<br>
[{<0.290.0>,94360,<br>
[ssl_manager,<br>
{current_function,{ssl_manager,handle_cast,2}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.13589.3>,24640,<br>
[{current_function,{ssl_manager,validate_session,3}},<br>
{initial_call,{ssl_manager,init_session_validator,1}}]},<br>
{<0.291.0>,12832,<br>
[tls_connection_sup,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.13590.3>,10960,<br>
[{current_function,{gen,do_call,4}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.318.0>,7904,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.308.0>,3056,<br>
[{current_function,{ranch_conns_sup,loop,4}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.385.0>,1872,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.254.0>,0,<br>
[code_server,<br>
{current_function,{code_server,loop,1}},<br>
{initial_call,{erlang,apply,2}}]},<br>
{<0.325.0>,0,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.357.0>,0,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]}]<br>
<br>
6> recon:proc_window(reductions, 10, 500).<br>
[{<0.290.0>,232358,<br>
[ssl_manager,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.291.0>,3058,<br>
[tls_connection_sup,<br>
{current_function,{gen_server,loop,6}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.308.0>,1129,<br>
[{current_function,{ranch_conns_sup,loop,4}},<br>
{initial_call,{proc_lib,init_p,5}}]},<br>
{<0.339.0>,435,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.340.0>,430,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.355.0>,430,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.353.0>,430,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.334.0>,430,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.366.0>,430,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]},<br>
{<0.360.0>,430,<br>
[{current_function,{prim_inet,accept0,2}},<br>
{initial_call,{ranch_acceptor,loop,3}}]}]<br>
<br>
RES in htop ~900-1000M<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br></div></div>
<mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>>><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
<br>
<br>
</blockquote><span class="">
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
</span><a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div><br></div>