[erlang-questions] 'ssl_otp_session_cache' ets table
Bogdan Andu
bog495@REDACTED
Mon Nov 10 10:54:49 CET 2014
only for R17 I tested this .
if you do not do cache at all the cpu usage increases
and dummy cache module is out of questions.
so a balance must be founded between session reuse (which a re good) and
processor usage.
application:get_env(ssl, session_lifetime).
returns ok the values.
If I have time I'll dig into the code to see what happens deeper.
On Fri, Nov 7, 2014 at 4:20 PM, Ingela Andin <ingela.andin@REDACTED> wrote:
> Hi!
>
>
> The following release note is from the first R15 version of ssl:
>
> "Invalidation handling of sessions could cause the time_stamp field in the
> session record to be set to undefined crashing the session clean up
> process. This did not affect the connections but would result in that the
> session table would grow."
>
>
> 2014-11-07 11:58 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>
>> I work with R14B04 as production and OTP 17.3 as experimental.
>>
>> R14B04 also deletes all entries after 24 hours.
>>
>> 17.3 also deletes all entries after 24 hours.
>>
>> I think is it has to do with passing the environment parameter like this:
>>
>> erl -ssl session_lifetime 60 ...
>>
>> I tried many variants and may be I miss something.
>>
>> may be I am mistaken but it is possible that only the default value, 24
>> hours,
>> is only considered as the environment variable is not passed correctly by
>> me.
>>
>> i think the environment parameters are the problem here because maybe i
>> specify them incorrctly.
>>
>> could you advise what is the proper method of correctly specifying
>> session_lifetime, because according to
>> manual is like :
>>
>> erl -ssl session_lifetime 60 ...
>>
>
>
> You can check with application:get_env(ssl, session_lifetime). to see if
> you succeed setting the variable.
>
>
>
>>
>> a ssl session must expire afte 60 seconds + delayed 60 seconds = 2
>> minutes.
>>
>> intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes,
>> they remain 24 hours .
>>
>> after 22 hours I have:
>>
>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP
>> +A:10
>> Time: local time 12:50:23, up for 000:21:54:49, 0ms latency,
>> Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k
>> allocated)
>> Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets
>> 720.6k
>>
>>
>> and for ets:i().
>>
>> ...........
>> 28695 ssl_otp_session_cache ordered_set 115 7134
>> ssl_manager
>> ..........
>>
>> The vm was configured with 60 seconds session lifetime and the client was
>> running 60 seconds
>> an I expected after 2 minutes to see less entries in the table but they
>> remained untouched
>>
>>
>> I will see what happens after 24 hours mark and I'll let you know
>>
>> and another thing...
>>
>> even if i pass the options:
>> {reuse_session, fun(_,_,_,_) -> false end},
>> {reuse_sessions, false},
>>
>> both or individually , to the ssl:listen/2 function the table is filled
>> with sessions and they expire after 24 hours,
>> only that cpu is working harder, 7-8% which means the sessions a not
>> reused and the ssl
>> handshake is happening every time.
>>
>>
> Was that for both R14 and R17?
>
>
> Regards Ingela Erlang/OTP team - Ericsson AB
>
>
>
>
>
>>
>> Bogdan
>>
>> On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin <ingela.andin@REDACTED>
>> wrote:
>>
>>> Hi!
>>>
>>> Which version of the ssl application are you using? The session cleanup
>>> was broken in one of the older versions.
>>>
>>> Regards Ingela Erlang/OTP - team Ericsson AB
>>>
>>>
>>> 2014-11-06 17:20 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>
>>>> yes,
>>>>
>>>> in function ssl_manager:invalidate_session/4, the call
>>>>
>>>> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}),
>>>> does the actual deletion of session from cache
>>>>
>>>> where delay_time/0 function returns ?CLEAN_SESSION_DB by default,
>>>> which is 60 seconds
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin <ingela.andin@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>>
>>>>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> i have tried to pass to execute th following forms of commands:
>>>>>>
>>>>>> erl -ssl session_lifetime 60
>>>>>>
>>>>>> erl -ssl session_lifetime '60'
>>>>>>
>>>>>> erl -ssl session_lifetime '[60]'
>>>>>>
>>>>>>
>>>>>> in order to set a ssl session_lifetime of 60 seconds.
>>>>>>
>>>>>> in init of ssl_manager , after session_lifetime + 5 seconds a process
>>>>>> is created to validate or invalidate sessions.
>>>>>> So, after 65 seconds a foldl is called on sessions table to check
>>>>>> sessions after formula:
>>>>>> Now - TimeStamp < LifeTime
>>>>>>
>>>>>> However, even after I stop the client, the table
>>>>>> 'ssl_otp_session_cache' 's size remains the same.
>>>>>>
>>>>>>
>>>>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the
>>>>>> table is sweeped to delete any expired sessions,
>>>>>>
>>>>>> but also nothing happens.
>>>>>>
>>>>>>
>>>>> There is also a delay in the actual deletion. In the first sweep the
>>>>> sessions will only be invalidated as there may already be
>>>>> spawned connection handlers that needs to read the session data before
>>>>> we may delete it. It is a performance trade off.
>>>>>
>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin <ingela.andin@REDACTED
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>>
>>>>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> on my production servers I have relayd (on OpenBSD) daemon as a
>>>>>>>> reverse proxy to some webservers
>>>>>>>> where one can fine tune some connection parameters, as well as some
>>>>>>>> ssl parameters.
>>>>>>>>
>>>>>>>> I give a snippet from a relayd.conf configuration file on one of my
>>>>>>>> production server:
>>>>>>>>
>>>>>>>> .............
>>>>>>>> #
>>>>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL
>>>>>>>> acceleration
>>>>>>>> #
>>>>>>>> http protocol www_ssl_prot {
>>>>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For"
>>>>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to
>>>>>>>> "X-Forwarded-By"
>>>>>>>> header change "Connection" to "close"
>>>>>>>>
>>>>>>>> response header change "Server" to "Apache 0.1"
>>>>>>>>
>>>>>>>> # Various TCP performance options
>>>>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 }
>>>>>>>>
>>>>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" }
>>>>>>>> ssl session cache disable
>>>>>>>> }
>>>>>>>>
>>>>>>>> the last directive tells relayd not to use ssl cache.
>>>>>>>>
>>>>>>>> This configuration is working for years and relayd was restarted
>>>>>>>> once by accident - my fault.
>>>>>>>>
>>>>>>>> SO, y question is:
>>>>>>>>
>>>>>>>> can we have this configurable in Erlang, in other words, we might
>>>>>>>> be able to start an erlang vm such as:
>>>>>>>>
>>>>>>>> erl -ssl session_cache 'disable' -name x@REDACTED ....
>>>>>>>>
>>>>>>>> The ssl option session_cache can be set to disabled by default and
>>>>>>>> can take values either disable or enabled.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> You can already disable the reuse of the sessions using the server
>>>>>>> option *{reuse_sessions, boolean()}* which default to true.
>>>>>>> The thing we plan to do is to have a configurable limit on the table
>>>>>>> size when sessions are reused.
>>>>>>>
>>>>>>> Regards Ingela Erlang/OTP team Ericsson AB
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Just to let you know...
>>>>>>>>
>>>>>>>> The statistics of a node running from yesterday when I opened this
>>>>>>>> thread of discussion, using a single client:
>>>>>>>>
>>>>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0)
>>>>>>>> CPU:2 SMP +A:10
>>>>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency,
>>>>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k
>>>>>>>> allocated)
>>>>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k,
>>>>>>>> Ets 80739.2k
>>>>>>>>
>>>>>>>> ...........
>>>>>>>>
>>>>>>>> So, can be this made configurable?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin <
>>>>>>>> ingela.andin@REDACTED> wrote:
>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall <tony@REDACTED>:
>>>>>>>>>
>>>>>>>>>> I think the ssl session times is the problem here, and the lack
>>>>>>>>>> of a maximum size.
>>>>>>>>>>
>>>>>>>>>> You can change the session time in the ssl environment:
>>>>>>>>>> session_lifetime
>>>>>>>>>> The default is set to 24 hours (in seconds) (if I read it
>>>>>>>>>> correctly, in ssl_manager.erl)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Default values are always hard. It is the maximum recommended time
>>>>>>>>> for a session to live
>>>>>>>>> according to the spec.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I guess that a session_cache_size could be a nice thing to have,
>>>>>>>>>> limiting the growth of the session cache.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Yes I agree.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> In other words you have to estimate the life time of your clients
>>>>>>>>>> and
>>>>>>>>>> try to find a reasonable session_lifetime to match that, without
>>>>>>>>>> blowing up
>>>>>>>>>> the system.
>>>>>>>>>>
>>>>>>>>>> Maybe the ssl_session_cache_api could be used to implement a
>>>>>>>>>> strategy with a max size
>>>>>>>>>> cache. Retire session least recently used, while performing the
>>>>>>>>>> update?
>>>>>>>>>> There is a time_stamp in the session that that could be used for
>>>>>>>>>> this purpose.
>>>>>>>>>>
>>>>>>>>>> OTP: Why is a fixed limit not implemented in the standard
>>>>>>>>>> ssl_session_cache?
>>>>>>>>>> Could this be a target for DOS attacks?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We are aware of the problem, and it is on our todo list. One
>>>>>>>>> reason it has not had top priority is that on the server side there are
>>>>>>>>> often other mechanisms
>>>>>>>>> like firewalls and webbserver settings that limits the problem.
>>>>>>>>> And the reason why it was not implemented in the first place is that
>>>>>>>>> implementations by nature
>>>>>>>>> are iterative and at first you are faced with a lot of bigger
>>>>>>>>> problems to solve and then you need to iterate and fine tune and fix things
>>>>>>>>> that you now have a better understanding of.
>>>>>>>>>
>>>>>>>>> In current master there is a change to the session table that
>>>>>>>>> limits the growth on the client side if the client behaves inappropriate.
>>>>>>>>> It also splits the session table into
>>>>>>>>> a server and a client table which is a better implementation as
>>>>>>>>> the same Erlang node can be both a client and a server at the same time. So
>>>>>>>>> if someone feels like contributing a max limit please base it on the
>>>>>>>>> master branch, otherwise I suspect someone compiling about it did raise
>>>>>>>>> the priority level a little.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> /Tony
>>>>>>>>>>
>>>>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu <bog495@REDACTED> wrote:
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Hi,
>>>>>>>>>> >
>>>>>>>>>> > I performed a series of test regarding the an Erlang SSL server.
>>>>>>>>>> >
>>>>>>>>>> > In this setup a major role is played by the table called
>>>>>>>>>> 'ssl_otp_session_cache', and of course the processes using it.
>>>>>>>>>> >
>>>>>>>>>> > The problem is that the size of table increases constantly and,
>>>>>>>>>> because an ets table does not automatically deallocate memory unless the
>>>>>>>>>> object are deleted from that table, the size of table remains the same even
>>>>>>>>>> if there no ssl connections to server.
>>>>>>>>>> >
>>>>>>>>>> > For example, with a single client running 'ad infinitum' the
>>>>>>>>>> table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate
>>>>>>>>>> around 60 MB of memory only for this table.
>>>>>>>>>> >
>>>>>>>>>> > Some info about this:
>>>>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie
>>>>>>>>>> operator
>>>>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2]
>>>>>>>>>> [async-threads:10] [kernel-poll:false]
>>>>>>>>>> >
>>>>>>>>>> > Eshell V6.2 (abort with ^G)
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > (n1@REDACTED)1> ets:i().
>>>>>>>>>> > id name type size mem owner
>>>>>>>>>> >
>>>>>>>>>> ----------------------------------------------------------------------------
>>>>>>>>>> > 12 cookies set 0 291 auth
>>>>>>>>>> > 4111 code set 410 26132
>>>>>>>>>> code_server
>>>>>>>>>> > 8208 code_names set 58 7459
>>>>>>>>>> code_server
>>>>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0
>>>>>>>>>> 291 httpc_manager
>>>>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291
>>>>>>>>>> ssl_manager
>>>>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291
>>>>>>>>>> ssl_manager
>>>>>>>>>> > 24598 ssl_otp_pem_cache set 3 360
>>>>>>>>>> ssl_manager
>>>>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057
>>>>>>>>>> 8421893 ssl_manager
>>>>>>>>>> > 32797 dets duplicate_bag 2 308
>>>>>>>>>> dets
>>>>>>>>>> > 40990 ign_requests set 0 291
>>>>>>>>>> inet_gethost_native
>>>>>>>>>> > 45087 ign_req_index set 0 291
>>>>>>>>>> inet_gethost_native
>>>>>>>>>> > 2261635104 shell_records ordered_set 0 81
>>>>>>>>>> <0.30638.37>
>>>>>>>>>> > ac_tab ac_tab set 33 2216
>>>>>>>>>> application_controller
>>>>>>>>>> > code_map code_map set 100 2791
>>>>>>>>>> <0.72.0>
>>>>>>>>>> > config config set 12 892
>>>>>>>>>> <0.72.0>
>>>>>>>>>> > dets_owners dets_owners set 1 298 dets
>>>>>>>>>> > dets_registry dets_registry set 1 299 dets
>>>>>>>>>> > file_io_servers file_io_servers set 1 344
>>>>>>>>>> file_server_2
>>>>>>>>>> > global_locks global_locks set 0 291
>>>>>>>>>> global_name_server
>>>>>>>>>> > global_names global_names set 0 291
>>>>>>>>>> global_name_server
>>>>>>>>>> > global_names_ext global_names_ext set 0 291
>>>>>>>>>> global_name_server
>>>>>>>>>> > global_pid_ids global_pid_ids bag 0 291
>>>>>>>>>> global_name_server
>>>>>>>>>> > global_pid_names global_pid_names bag 0 291
>>>>>>>>>> global_name_server
>>>>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0
>>>>>>>>>> 291 httpc_manager
>>>>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0
>>>>>>>>>> 291 httpc_manager
>>>>>>>>>> > inet_cache inet_cache bag 0 291 inet_db
>>>>>>>>>> > inet_db inet_db set 29 600 inet_db
>>>>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291
>>>>>>>>>> inet_db
>>>>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291
>>>>>>>>>> inet_db
>>>>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0
>>>>>>>>>> 291 inet_db
>>>>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0
>>>>>>>>>> 291 inet_db
>>>>>>>>>> > models models set 3 28952
>>>>>>>>>> <0.72.0>
>>>>>>>>>> > sys_dist sys_dist set 1 334
>>>>>>>>>> net_kernel
>>>>>>>>>> > ok
>>>>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache).
>>>>>>>>>> > undefined
>>>>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024.
>>>>>>>>>> > 65796.0390625
>>>>>>>>>> > (n1@REDACTED)8> memory().
>>>>>>>>>> > [{total,92699464},
>>>>>>>>>> > {processes,8964000},
>>>>>>>>>> > {processes_used,8963152},
>>>>>>>>>> > {system,83735464},
>>>>>>>>>> > {atom,429569},
>>>>>>>>>> > {atom_used,421768},
>>>>>>>>>> > {binary,199040},
>>>>>>>>>> > {code,10411520},
>>>>>>>>>> > {ets,69163032}]
>>>>>>>>>> >
>>>>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is
>>>>>>>>>> roughly 64 MB in 12 hours.
>>>>>>>>>> >
>>>>>>>>>> > On an OpenBSD platform such process gets killed immediately it
>>>>>>>>>> hits some memory and/or CPU limits.
>>>>>>>>>> >
>>>>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory,
>>>>>>>>>> otherwise the Erlang VM would be killed.
>>>>>>>>>> >
>>>>>>>>>> > How can one control , tweak or configure this table such that
>>>>>>>>>> it does not accumulate such data at such high rate.
>>>>>>>>>> >
>>>>>>>>>> > I seems the table being created private, and there is no way to
>>>>>>>>>> ets:delete_all_objects/1 from table manually.
>>>>>>>>>> >
>>>>>>>>>> > I know that this table caches some SSL data related to clients,
>>>>>>>>>> but the client has the same IP address,
>>>>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection
>>>>>>>>>> info about the same client when only the ephemeral peer port
>>>>>>>>>> > differs?
>>>>>>>>>> >
>>>>>>>>>> > How the size of this table can be held in reasonable limits and
>>>>>>>>>> the rate it's size increases ?
>>>>>>>>>> >
>>>>>>>>>> > Please if somebody shed some light on these issues.
>>>>>>>>>> >
>>>>>>>>>> > Thank you,
>>>>>>>>>> >
>>>>>>>>>> > Bogdan
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > _______________________________________________
>>>>>>>>>> > erlang-questions mailing list
>>>>>>>>>> > erlang-questions@REDACTED
>>>>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>>>
>>>>>>>>>> "Installing applications can lead to corruption over time.
>>>>>>>>>> Applications gradually write over each other's libraries, partial upgrades
>>>>>>>>>> occur, user and system errors happen, and minute changes may be
>>>>>>>>>> unnoticeable and difficult to fix"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> erlang-questions mailing list
>>>>>>>>>> erlang-questions@REDACTED
>>>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141110/34b251e4/attachment.htm>
More information about the erlang-questions
mailing list