[erlang-questions] 'ssl_otp_session_cache' ets table
Ingela Andin
ingela.andin@REDACTED
Fri Nov 7 09:37:32 CET 2014
Hi!
Which version of the ssl application are you using? The session cleanup was
broken in one of the older versions.
Regards Ingela Erlang/OTP - team Ericsson AB
2014-11-06 17:20 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
> yes,
>
> in function ssl_manager:invalidate_session/4, the call
>
> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}),
> does the actual deletion of session from cache
>
> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which
> is 60 seconds
>
>
>
>
> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin <ingela.andin@REDACTED>
> wrote:
>
>> Hi!
>>
>>
>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>
>>> Hi,
>>>
>>> i have tried to pass to execute th following forms of commands:
>>>
>>> erl -ssl session_lifetime 60
>>>
>>> erl -ssl session_lifetime '60'
>>>
>>> erl -ssl session_lifetime '[60]'
>>>
>>>
>>> in order to set a ssl session_lifetime of 60 seconds.
>>>
>>> in init of ssl_manager , after session_lifetime + 5 seconds a process is
>>> created to validate or invalidate sessions.
>>> So, after 65 seconds a foldl is called on sessions table to check
>>> sessions after formula:
>>> Now - TimeStamp < LifeTime
>>>
>>> However, even after I stop the client, the table 'ssl_otp_session_cache'
>>> 's size remains the same.
>>>
>>>
>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table
>>> is sweeped to delete any expired sessions,
>>>
>>> but also nothing happens.
>>>
>>>
>> There is also a delay in the actual deletion. In the first sweep the
>> sessions will only be invalidated as there may already be
>> spawned connection handlers that needs to read the session data before we
>> may delete it. It is a performance trade off.
>>
>> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>>
>>
>>>
>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin <ingela.andin@REDACTED>
>>> wrote:
>>>
>>>> Hi!
>>>>
>>>>
>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>>
>>>>> Hi,
>>>>>
>>>>> on my production servers I have relayd (on OpenBSD) daemon as a
>>>>> reverse proxy to some webservers
>>>>> where one can fine tune some connection parameters, as well as some
>>>>> ssl parameters.
>>>>>
>>>>> I give a snippet from a relayd.conf configuration file on one of my
>>>>> production server:
>>>>>
>>>>> .............
>>>>> #
>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL
>>>>> acceleration
>>>>> #
>>>>> http protocol www_ssl_prot {
>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For"
>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
>>>>> header change "Connection" to "close"
>>>>>
>>>>> response header change "Server" to "Apache 0.1"
>>>>>
>>>>> # Various TCP performance options
>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 }
>>>>>
>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" }
>>>>> ssl session cache disable
>>>>> }
>>>>>
>>>>> the last directive tells relayd not to use ssl cache.
>>>>>
>>>>> This configuration is working for years and relayd was restarted once
>>>>> by accident - my fault.
>>>>>
>>>>> SO, y question is:
>>>>>
>>>>> can we have this configurable in Erlang, in other words, we might be
>>>>> able to start an erlang vm such as:
>>>>>
>>>>> erl -ssl session_cache 'disable' -name x@REDACTED ....
>>>>>
>>>>> The ssl option session_cache can be set to disabled by default and can
>>>>> take values either disable or enabled.
>>>>>
>>>>>
>>>>>
>>>> You can already disable the reuse of the sessions using the server
>>>> option *{reuse_sessions, boolean()}* which default to true.
>>>> The thing we plan to do is to have a configurable limit on the table
>>>> size when sessions are reused.
>>>>
>>>> Regards Ingela Erlang/OTP team Ericsson AB
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Just to let you know...
>>>>>
>>>>> The statistics of a node running from yesterday when I opened this
>>>>> thread of discussion, using a single client:
>>>>>
>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2
>>>>> SMP +A:10
>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency,
>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k
>>>>> allocated)
>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k,
>>>>> Ets 80739.2k
>>>>>
>>>>> ...........
>>>>>
>>>>> So, can be this made configurable?
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall <tony@REDACTED>:
>>>>>>
>>>>>>> I think the ssl session times is the problem here, and the lack of a
>>>>>>> maximum size.
>>>>>>>
>>>>>>> You can change the session time in the ssl environment:
>>>>>>> session_lifetime
>>>>>>> The default is set to 24 hours (in seconds) (if I read it correctly,
>>>>>>> in ssl_manager.erl)
>>>>>>>
>>>>>>
>>>>>> Default values are always hard. It is the maximum recommended time
>>>>>> for a session to live
>>>>>> according to the spec.
>>>>>>
>>>>>>
>>>>>>> I guess that a session_cache_size could be a nice thing to have,
>>>>>>> limiting the growth of the session cache.
>>>>>>>
>>>>>>>
>>>>>> Yes I agree.
>>>>>>
>>>>>>
>>>>>>> In other words you have to estimate the life time of your clients and
>>>>>>> try to find a reasonable session_lifetime to match that, without
>>>>>>> blowing up
>>>>>>> the system.
>>>>>>>
>>>>>>> Maybe the ssl_session_cache_api could be used to implement a
>>>>>>> strategy with a max size
>>>>>>> cache. Retire session least recently used, while performing the
>>>>>>> update?
>>>>>>> There is a time_stamp in the session that that could be used for
>>>>>>> this purpose.
>>>>>>>
>>>>>>> OTP: Why is a fixed limit not implemented in the standard
>>>>>>> ssl_session_cache?
>>>>>>> Could this be a target for DOS attacks?
>>>>>>>
>>>>>>
>>>>>>
>>>>>> We are aware of the problem, and it is on our todo list. One reason
>>>>>> it has not had top priority is that on the server side there are often
>>>>>> other mechanisms
>>>>>> like firewalls and webbserver settings that limits the problem. And
>>>>>> the reason why it was not implemented in the first place is that
>>>>>> implementations by nature
>>>>>> are iterative and at first you are faced with a lot of bigger
>>>>>> problems to solve and then you need to iterate and fine tune and fix things
>>>>>> that you now have a better understanding of.
>>>>>>
>>>>>> In current master there is a change to the session table that limits
>>>>>> the growth on the client side if the client behaves inappropriate. It also
>>>>>> splits the session table into
>>>>>> a server and a client table which is a better implementation as the
>>>>>> same Erlang node can be both a client and a server at the same time. So if
>>>>>> someone feels like contributing a max limit please base it on the master
>>>>>> branch, otherwise I suspect someone compiling about it did raise the
>>>>>> priority level a little.
>>>>>>
>>>>>>
>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>
>>>>>>
>>>>>>
>>>>>>> /Tony
>>>>>>>
>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu <bog495@REDACTED> wrote:
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I performed a series of test regarding the an Erlang SSL server.
>>>>>>> >
>>>>>>> > In this setup a major role is played by the table called
>>>>>>> 'ssl_otp_session_cache', and of course the processes using it.
>>>>>>> >
>>>>>>> > The problem is that the size of table increases constantly and,
>>>>>>> because an ets table does not automatically deallocate memory unless the
>>>>>>> object are deleted from that table, the size of table remains the same even
>>>>>>> if there no ssl connections to server.
>>>>>>> >
>>>>>>> > For example, with a single client running 'ad infinitum' the table
>>>>>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around
>>>>>>> 60 MB of memory only for this table.
>>>>>>> >
>>>>>>> > Some info about this:
>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie
>>>>>>> operator
>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2]
>>>>>>> [async-threads:10] [kernel-poll:false]
>>>>>>> >
>>>>>>> > Eshell V6.2 (abort with ^G)
>>>>>>> >
>>>>>>> >
>>>>>>> > (n1@REDACTED)1> ets:i().
>>>>>>> > id name type size mem owner
>>>>>>> >
>>>>>>> ----------------------------------------------------------------------------
>>>>>>> > 12 cookies set 0 291 auth
>>>>>>> > 4111 code set 410 26132
>>>>>>> code_server
>>>>>>> > 8208 code_names set 58 7459
>>>>>>> code_server
>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0
>>>>>>> 291 httpc_manager
>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291
>>>>>>> ssl_manager
>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291
>>>>>>> ssl_manager
>>>>>>> > 24598 ssl_otp_pem_cache set 3 360
>>>>>>> ssl_manager
>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893
>>>>>>> ssl_manager
>>>>>>> > 32797 dets duplicate_bag 2 308
>>>>>>> dets
>>>>>>> > 40990 ign_requests set 0 291
>>>>>>> inet_gethost_native
>>>>>>> > 45087 ign_req_index set 0 291
>>>>>>> inet_gethost_native
>>>>>>> > 2261635104 shell_records ordered_set 0 81
>>>>>>> <0.30638.37>
>>>>>>> > ac_tab ac_tab set 33 2216
>>>>>>> application_controller
>>>>>>> > code_map code_map set 100 2791 <0.72.0>
>>>>>>> > config config set 12 892 <0.72.0>
>>>>>>> > dets_owners dets_owners set 1 298 dets
>>>>>>> > dets_registry dets_registry set 1 299 dets
>>>>>>> > file_io_servers file_io_servers set 1 344
>>>>>>> file_server_2
>>>>>>> > global_locks global_locks set 0 291
>>>>>>> global_name_server
>>>>>>> > global_names global_names set 0 291
>>>>>>> global_name_server
>>>>>>> > global_names_ext global_names_ext set 0 291
>>>>>>> global_name_server
>>>>>>> > global_pid_ids global_pid_ids bag 0 291
>>>>>>> global_name_server
>>>>>>> > global_pid_names global_pid_names bag 0 291
>>>>>>> global_name_server
>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0
>>>>>>> 291 httpc_manager
>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0
>>>>>>> 291 httpc_manager
>>>>>>> > inet_cache inet_cache bag 0 291 inet_db
>>>>>>> > inet_db inet_db set 29 600 inet_db
>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db
>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db
>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291
>>>>>>> inet_db
>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291
>>>>>>> inet_db
>>>>>>> > models models set 3 28952 <0.72.0>
>>>>>>> > sys_dist sys_dist set 1 334 net_kernel
>>>>>>> > ok
>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache).
>>>>>>> > undefined
>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024.
>>>>>>> > 65796.0390625
>>>>>>> > (n1@REDACTED)8> memory().
>>>>>>> > [{total,92699464},
>>>>>>> > {processes,8964000},
>>>>>>> > {processes_used,8963152},
>>>>>>> > {system,83735464},
>>>>>>> > {atom,429569},
>>>>>>> > {atom_used,421768},
>>>>>>> > {binary,199040},
>>>>>>> > {code,10411520},
>>>>>>> > {ets,69163032}]
>>>>>>> >
>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly
>>>>>>> 64 MB in 12 hours.
>>>>>>> >
>>>>>>> > On an OpenBSD platform such process gets killed immediately it
>>>>>>> hits some memory and/or CPU limits.
>>>>>>> >
>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory,
>>>>>>> otherwise the Erlang VM would be killed.
>>>>>>> >
>>>>>>> > How can one control , tweak or configure this table such that it
>>>>>>> does not accumulate such data at such high rate.
>>>>>>> >
>>>>>>> > I seems the table being created private, and there is no way to
>>>>>>> ets:delete_all_objects/1 from table manually.
>>>>>>> >
>>>>>>> > I know that this table caches some SSL data related to clients,
>>>>>>> but the client has the same IP address,
>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection
>>>>>>> info about the same client when only the ephemeral peer port
>>>>>>> > differs?
>>>>>>> >
>>>>>>> > How the size of this table can be held in reasonable limits and
>>>>>>> the rate it's size increases ?
>>>>>>> >
>>>>>>> > Please if somebody shed some light on these issues.
>>>>>>> >
>>>>>>> > Thank you,
>>>>>>> >
>>>>>>> > Bogdan
>>>>>>> >
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > erlang-questions mailing list
>>>>>>> > erlang-questions@REDACTED
>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>
>>>>>>> "Installing applications can lead to corruption over time.
>>>>>>> Applications gradually write over each other's libraries, partial upgrades
>>>>>>> occur, user and system errors happen, and minute changes may be
>>>>>>> unnoticeable and difficult to fix"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> erlang-questions mailing list
>>>>>>> erlang-questions@REDACTED
>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141107/fbcb4693/attachment.htm>
More information about the erlang-questions
mailing list