[erlang-questions] Memory consumption in ssl application (OTP-18.3.1)
Sereysethy TOUCH
touch.sereysethy@REDACTED
Thu Apr 28 16:53:07 CEST 2016
Hi,
I used to face this kind of problem when I updated erlang from OTP 17.x.x
to 18 but I think the problem was fixed already in 18.2
Here is what I found out:
My problem was that `ets` table keeps growing and it is caused by
`server_ssl_otp_session_cache`
I traced it back to the source code of Erlang 17 & Erlang 18 release
branch, in ssl module/ssl_manager.erl. There was a mistake in calculation
of 24H_in_msec in Erlang 17.
`Erlang OTP 17`
-define('24H_in_msec', 8640000).
-define('24H_in_sec', 8640).
-define(GEN_UNIQUE_ID_MAX_TRIES, 10).
-define(SESSION_VALIDATION_INTERVAL, 60000).
-define(CLEAR_PEM_CACHE, 120000).
-define(CLEAN_SESSION_DB, 60000).
-define(CLEAN_CERT_DB, 500).
-define(NOT_TO_BIG, 10).
`Erlang OTP 18`
-define('24H_in_msec', 86400000).
-define('24H_in_sec', 86400).
-define(GEN_UNIQUE_ID_MAX_TRIES, 10).
-define(SESSION_VALIDATION_INTERVAL, 60000).
-define(CLEAR_PEM_CACHE, 120000).
-define(CLEAN_SESSION_DB, 60000).
-define(CLEAN_CERT_DB, 500).
-define(NOT_TO_BIG, 10).
So you can try to pass `session_lifetime` to 30mn (1800 seconds) to `ssl`
application (module).
Sethy
On Thu, Apr 28, 2016 at 5:36 PM, <vicbaz@REDACTED> wrote:
> Hello,
>
> Could you please clarify one moment. Is the ssl session cache used only
> when reuse_sessions is set to true? Is there a way to turn off the cache
> completely? I'm trying to figure out what settings are needed for
> thousands of short-lived connections.
>
> 28/04/16 12:21, Ingela Andin пишет:
>
>> Hi!
>>
>>
>> 2016-04-27 15:44 GMT+02:00 <vicbaz@REDACTED <mailto:vicbaz@REDACTED>>:
>>
>> Hello,
>>
>> The reason was session_cache_server_max (defaults to 1000).
>> The ssl_manager quickly get filled by
>>
>> {'$gen_cast',{invalidate_session,8443,{session,<<...>>,...}}}
>> {delayed_clean_session,{8443,<<14,50,229,...>>},40986}
>>
>> messages.
>>
>> I see, we will looking into mitigating this effect.
>>
>> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>>
>>
>> 27/04/16 13:29, Ingela Andin пишет:
>>
>> Hi!
>>
>> You could try using the observer application to find out more
>> information about your system state.
>> There is no apparent reason why ssl-7.3 should consume a lot
>> more memory
>> than ssl-6.0.1.2.
>> But a lot has append between the versions and heavy load may
>> find corner
>> cases that where missed.
>>
>> Regards Ingela Erlang/OTP Team - Ericsson AB
>>
>>
>> 2016-04-25 14:35 GMT+02:00 <vicbaz@REDACTED
>> <mailto:vicbaz@REDACTED> <mailto:vicbaz@REDACTED
>>
>> <mailto:vicbaz@REDACTED>>>:
>>
>>
>> Hello,
>>
>> When I run
>>
>> https://github.com/ninenines/cowboy/tree/1.0.4/examples/ssl_hello_world
>> under load (using https://github.com/JoeDog/siege) I found
>> a big
>> difference in memory consumption between OTP-17.5 and
>> OTP-18.3. I know
>> the ssl application has many changes in new versions. Is
>> this memory
>> usage normal now or can be reduced using some ssl
>> application settings?
>>
>> I use OTP-17.5.6.9 and OTP-18.3.1.
>>
>>
>>
>> ================================================================================
>> Erlang/OTP 17 [erts-6.4.1.6] [source] [64-bit] [smp:2:2]
>> [async-threads:10] [hipe] [kernel-poll:false]
>>
>> 1> application:which_applications().
>> [{ssl_hello_world,"Cowboy Hello World example with SSL.",
>> "1"},
>> {cowboy,"Small, fast, modular HTTP server.","1.0.4"},
>> {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
>> {cowlib,"Support library for manipulating Web protocols.",
>> "1.0.2"},
>> {ssl,"Erlang/OTP SSL application","6.0.1.2"},
>> {public_key,"Public key infrastructure","0.23"},
>> {crypto,"CRYPTO","3.5"},
>> {asn1,"The Erlang ASN1 compiler version 3.0.4","3.0.4"},
>> {stdlib,"ERTS CXC 138 10","2.4"},
>> {kernel,"ERTS CXC 138 10","3.2.0.1"}]
>>
>> 2> erlang:memory().
>> [{total,43913840},
>> {processes,8036312},
>> {processes_used,8030744},
>> {system,35877528},
>> {atom,470537},
>> {atom_used,455904},
>> {binary,1498800},
>> {code,11074476},
>> {ets,14516424}]
>>
>> 3> recon_alloc:memory(used).
>> 45519464
>>
>> 4> recon_alloc:memory(usage).
>> 0.7212344918526264
>>
>> 5> recon:proc_window(memory, 10, 500).
>> [{<0.25910.2>,68120,
>> [{current_function,{gen_fsm,loop,7}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.25908.2>,68120,
>> [{current_function,{gen_fsm,loop,7}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.284.0>,15808,
>> [ssl_manager,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.285.0>,12712,
>> [tls_connection_sup,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.25909.2>,5888,
>> [{current_function,{gen,do_call,4}},
>> {initial_call,{cowboy_protocol,init,4}}]},
>> {<0.324.0>,4888,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.302.0>,4808,
>> [{current_function,{ranch_conns_sup,loop,4}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.330.0>,3016,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.25911.2>,2848,
>> [{current_function,{gen,do_call,4}},
>> {initial_call,{cowboy_protocol,init,4}}]},
>> {<0.340.0>,1872,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]}]
>>
>> 6> recon:proc_window(reductions, 10, 500).
>> [{<0.2634.3>,11161,
>> [{current_function,{gen_fsm,loop,7}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.2633.3>,9747,
>> [{current_function,{crypto,int_to_bin_pos,2}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.284.0>,4097,
>> [ssl_manager,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.285.0>,3314,
>> [tls_connection_sup,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.302.0>,1003,
>> [{current_function,{ranch_conns_sup,loop,4}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.378.0>,426,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.372.0>,425,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.379.0>,425,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.377.0>,425,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.343.0>,425,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]}]
>>
>> RES in htop ~50-60M
>>
>>
>>
>> ================================================================================
>>
>> Erlang/OTP 18 [erts-7.3.1] [source] [64-bit] [smp:2:2]
>> [async-threads:10] [hipe] [kernel-poll:false]
>>
>> 1> application:which_applications().
>> [{ssl_hello_world,"Cowboy Hello World example with SSL.",
>> "1"},
>> {cowboy,"Small, fast, modular HTTP server.","1.0.4"},
>> {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
>> {cowlib,"Support library for manipulating Web protocols.",
>> "1.0.2"},
>> {ssl,"Erlang/OTP SSL application","7.3"},
>> {public_key,"Public key infrastructure","1.1.1"},
>> {crypto,"CRYPTO","3.6.3"},
>> {asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"},
>> {recon,"Diagnostic tools for production use","2.3.1"},
>> {stdlib,"ERTS CXC 138 10","2.8"},
>> {kernel,"ERTS CXC 138 10","4.2"}]
>>
>> 2> erlang:memory().
>> [{total,949153560},
>> {processes,902969888},
>> {processes_used,902558312},
>> {system,46183672},
>> {atom,437761},
>> {atom_used,432875},
>> {binary,48320},
>> {code,11407283},
>> {ets,883640}]
>>
>> 3> recon_alloc:memory(used).
>> 863194560
>>
>> 4> recon_alloc:memory(usage).
>> 0.8005009562139471
>>
>> 5> recon:proc_window(memory, 10, 500).
>> [{<0.290.0>,94360,
>> [ssl_manager,
>> {current_function,{ssl_manager,handle_cast,2}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.13589.3>,24640,
>> [{current_function,{ssl_manager,validate_session,3}},
>> {initial_call,{ssl_manager,init_session_validator,1}}]},
>> {<0.291.0>,12832,
>> [tls_connection_sup,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.13590.3>,10960,
>> [{current_function,{gen,do_call,4}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.318.0>,7904,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.308.0>,3056,
>> [{current_function,{ranch_conns_sup,loop,4}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.385.0>,1872,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.254.0>,0,
>> [code_server,
>> {current_function,{code_server,loop,1}},
>> {initial_call,{erlang,apply,2}}]},
>> {<0.325.0>,0,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.357.0>,0,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]}]
>>
>> 6> recon:proc_window(reductions, 10, 500).
>> [{<0.290.0>,232358,
>> [ssl_manager,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.291.0>,3058,
>> [tls_connection_sup,
>> {current_function,{gen_server,loop,6}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.308.0>,1129,
>> [{current_function,{ranch_conns_sup,loop,4}},
>> {initial_call,{proc_lib,init_p,5}}]},
>> {<0.339.0>,435,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.340.0>,430,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.355.0>,430,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.353.0>,430,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.334.0>,430,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.366.0>,430,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]},
>> {<0.360.0>,430,
>> [{current_function,{prim_inet,accept0,2}},
>> {initial_call,{ranch_acceptor,loop,3}}]}]
>>
>> RES in htop ~900-1000M
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>> <mailto:erlang-questions@REDACTED
>> <mailto:erlang-questions@REDACTED>>
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>>
>> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160428/05cf59da/attachment.htm>
More information about the erlang-questions
mailing list