[erlang-questions] 'ssl_otp_session_cache' ets table
Ben Murphy
benmmurphy@REDACTED
Fri Nov 7 12:28:09 CET 2014
If you are desperate you can implement your own ssl session cache that
does nothing.
erl -ssl session_cb dummy_session_cache
https://github.com/mitadmin/ionstorm/blob/master/dummy_session_cache.erl
I think this was working for me with R1403.
On Fri, Nov 7, 2014 at 10:58 AM, Bogdan Andu <bog495@REDACTED> wrote:
> I work with R14B04 as production and OTP 17.3 as experimental.
>
> R14B04 also deletes all entries after 24 hours.
>
> 17.3 also deletes all entries after 24 hours.
>
> I think is it has to do with passing the environment parameter like this:
>
> erl -ssl session_lifetime 60 ...
>
> I tried many variants and may be I miss something.
>
> may be I am mistaken but it is possible that only the default value, 24
> hours,
> is only considered as the environment variable is not passed correctly by
> me.
>
> i think the environment parameters are the problem here because maybe i
> specify them incorrctly.
>
> could you advise what is the proper method of correctly specifying
> session_lifetime, because according to
> manual is like :
>
> erl -ssl session_lifetime 60 ...
>
> a ssl session must expire afte 60 seconds + delayed 60 seconds = 2 minutes.
>
> intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes,
> they remain 24 hours .
>
> after 22 hours I have:
>
> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP
> +A:10
> Time: local time 12:50:23, up for 000:21:54:49, 0ms latency,
> Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k allocated)
> Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets
> 720.6k
>
>
> and for ets:i().
>
> ...........
> 28695 ssl_otp_session_cache ordered_set 115 7134
> ssl_manager
> ..........
>
> The vm was configured with 60 seconds session lifetime and the client was
> running 60 seconds
> an I expected after 2 minutes to see less entries in the table but they
> remained untouched
>
>
> I will see what happens after 24 hours mark and I'll let you know
>
> and another thing...
>
> even if i pass the options:
> {reuse_session, fun(_,_,_,_) -> false end},
> {reuse_sessions, false},
>
> both or individually , to the ssl:listen/2 function the table is filled with
> sessions and they expire after 24 hours,
> only that cpu is working harder, 7-8% which means the sessions a not reused
> and the ssl
> handshake is happening every time.
>
>
> Bogdan
>
> On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin <ingela.andin@REDACTED>
> wrote:
>>
>> Hi!
>>
>> Which version of the ssl application are you using? The session cleanup
>> was broken in one of the older versions.
>>
>> Regards Ingela Erlang/OTP - team Ericsson AB
>>
>>
>> 2014-11-06 17:20 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>
>>> yes,
>>>
>>> in function ssl_manager:invalidate_session/4, the call
>>>
>>> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}),
>>> does the actual deletion of session from cache
>>>
>>> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which
>>> is 60 seconds
>>>
>>>
>>>
>>>
>>> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin <ingela.andin@REDACTED>
>>> wrote:
>>>>
>>>> Hi!
>>>>
>>>>
>>>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> i have tried to pass to execute th following forms of commands:
>>>>>
>>>>> erl -ssl session_lifetime 60
>>>>>
>>>>> erl -ssl session_lifetime '60'
>>>>>
>>>>> erl -ssl session_lifetime '[60]'
>>>>>
>>>>>
>>>>> in order to set a ssl session_lifetime of 60 seconds.
>>>>>
>>>>> in init of ssl_manager , after session_lifetime + 5 seconds a process
>>>>> is created to validate or invalidate sessions.
>>>>> So, after 65 seconds a foldl is called on sessions table to check
>>>>> sessions after formula:
>>>>> Now - TimeStamp < LifeTime
>>>>>
>>>>> However, even after I stop the client, the table
>>>>> 'ssl_otp_session_cache' 's size remains the same.
>>>>>
>>>>>
>>>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table
>>>>> is sweeped to delete any expired sessions,
>>>>>
>>>>> but also nothing happens.
>>>>>
>>>>
>>>> There is also a delay in the actual deletion. In the first sweep the
>>>> sessions will only be invalidated as there may already be
>>>> spawned connection handlers that needs to read the session data before
>>>> we may delete it. It is a performance trade off.
>>>>
>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>> wrote:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>>
>>>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> on my production servers I have relayd (on OpenBSD) daemon as a
>>>>>>> reverse proxy to some webservers
>>>>>>> where one can fine tune some connection parameters, as well as some
>>>>>>> ssl parameters.
>>>>>>>
>>>>>>> I give a snippet from a relayd.conf configuration file on one of my
>>>>>>> production server:
>>>>>>>
>>>>>>> .............
>>>>>>> #
>>>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL
>>>>>>> acceleration
>>>>>>> #
>>>>>>> http protocol www_ssl_prot {
>>>>>>> header append "$REMOTE_ADDR" to "X-Forwarded-For"
>>>>>>> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
>>>>>>> header change "Connection" to "close"
>>>>>>>
>>>>>>> response header change "Server" to "Apache 0.1"
>>>>>>>
>>>>>>> # Various TCP performance options
>>>>>>> tcp { nodelay, sack, socket buffer 65536, backlog 128 }
>>>>>>>
>>>>>>> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" }
>>>>>>> ssl session cache disable
>>>>>>> }
>>>>>>>
>>>>>>> the last directive tells relayd not to use ssl cache.
>>>>>>>
>>>>>>> This configuration is working for years and relayd was restarted once
>>>>>>> by accident - my fault.
>>>>>>>
>>>>>>> SO, y question is:
>>>>>>>
>>>>>>> can we have this configurable in Erlang, in other words, we might be
>>>>>>> able to start an erlang vm such as:
>>>>>>>
>>>>>>> erl -ssl session_cache 'disable' -name x@REDACTED ....
>>>>>>>
>>>>>>> The ssl option session_cache can be set to disabled by default and
>>>>>>> can take values either disable or enabled.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> You can already disable the reuse of the sessions using the server
>>>>>> option {reuse_sessions, boolean()} which default to true.
>>>>>> The thing we plan to do is to have a configurable limit on the table
>>>>>> size when sessions are reused.
>>>>>>
>>>>>> Regards Ingela Erlang/OTP team Ericsson AB
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Just to let you know...
>>>>>>>
>>>>>>> The statistics of a node running from yesterday when I opened this
>>>>>>> thread of discussion, using a single client:
>>>>>>>
>>>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2
>>>>>>> SMP +A:10
>>>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency,
>>>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k
>>>>>>> allocated)
>>>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k,
>>>>>>> Ets 80739.2k
>>>>>>>
>>>>>>> ...........
>>>>>>>
>>>>>>> So, can be this made configurable?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall <tony@REDACTED>:
>>>>>>>>>
>>>>>>>>> I think the ssl session times is the problem here, and the lack of
>>>>>>>>> a maximum size.
>>>>>>>>>
>>>>>>>>> You can change the session time in the ssl environment:
>>>>>>>>> session_lifetime
>>>>>>>>> The default is set to 24 hours (in seconds) (if I read it
>>>>>>>>> correctly, in ssl_manager.erl)
>>>>>>>>
>>>>>>>>
>>>>>>>> Default values are always hard. It is the maximum recommended time
>>>>>>>> for a session to live
>>>>>>>> according to the spec.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I guess that a session_cache_size could be a nice thing to have,
>>>>>>>>> limiting the growth of the session cache.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yes I agree.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> In other words you have to estimate the life time of your clients
>>>>>>>>> and
>>>>>>>>> try to find a reasonable session_lifetime to match that, without
>>>>>>>>> blowing up
>>>>>>>>> the system.
>>>>>>>>>
>>>>>>>>> Maybe the ssl_session_cache_api could be used to implement a
>>>>>>>>> strategy with a max size
>>>>>>>>> cache. Retire session least recently used, while performing the
>>>>>>>>> update?
>>>>>>>>> There is a time_stamp in the session that that could be used for
>>>>>>>>> this purpose.
>>>>>>>>>
>>>>>>>>> OTP: Why is a fixed limit not implemented in the standard
>>>>>>>>> ssl_session_cache?
>>>>>>>>> Could this be a target for DOS attacks?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We are aware of the problem, and it is on our todo list. One reason
>>>>>>>> it has not had top priority is that on the server side there are often other
>>>>>>>> mechanisms
>>>>>>>> like firewalls and webbserver settings that limits the problem. And
>>>>>>>> the reason why it was not implemented in the first place is that
>>>>>>>> implementations by nature
>>>>>>>> are iterative and at first you are faced with a lot of bigger
>>>>>>>> problems to solve and then you need to iterate and fine tune and fix things
>>>>>>>> that you now have a better understanding of.
>>>>>>>>
>>>>>>>> In current master there is a change to the session table that limits
>>>>>>>> the growth on the client side if the client behaves inappropriate. It also
>>>>>>>> splits the session table into
>>>>>>>> a server and a client table which is a better implementation as the
>>>>>>>> same Erlang node can be both a client and a server at the same time. So if
>>>>>>>> someone feels like contributing a max limit please base it on the master
>>>>>>>> branch, otherwise I suspect someone compiling about it did raise the
>>>>>>>> priority level a little.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> /Tony
>>>>>>>>>
>>>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu <bog495@REDACTED> wrote:
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Hi,
>>>>>>>>> >
>>>>>>>>> > I performed a series of test regarding the an Erlang SSL server.
>>>>>>>>> >
>>>>>>>>> > In this setup a major role is played by the table called
>>>>>>>>> > 'ssl_otp_session_cache', and of course the processes using it.
>>>>>>>>> >
>>>>>>>>> > The problem is that the size of table increases constantly and,
>>>>>>>>> > because an ets table does not automatically deallocate memory unless the
>>>>>>>>> > object are deleted from that table, the size of table remains the same even
>>>>>>>>> > if there no ssl connections to server.
>>>>>>>>> >
>>>>>>>>> > For example, with a single client running 'ad infinitum' the
>>>>>>>>> > table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate
>>>>>>>>> > around 60 MB of memory only for this table.
>>>>>>>>> >
>>>>>>>>> > Some info about this:
>>>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie
>>>>>>>>> > operator
>>>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2]
>>>>>>>>> > [async-threads:10] [kernel-poll:false]
>>>>>>>>> >
>>>>>>>>> > Eshell V6.2 (abort with ^G)
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > (n1@REDACTED)1> ets:i().
>>>>>>>>> > id name type size mem owner
>>>>>>>>> >
>>>>>>>>> > ----------------------------------------------------------------------------
>>>>>>>>> > 12 cookies set 0 291 auth
>>>>>>>>> > 4111 code set 410 26132
>>>>>>>>> > code_server
>>>>>>>>> > 8208 code_names set 58 7459
>>>>>>>>> > code_server
>>>>>>>>> > 12307 httpc_manager__session_cookie_db bag 0
>>>>>>>>> > 291 httpc_manager
>>>>>>>>> > 16404 ssl_otp_cacertificate_db set 0 291
>>>>>>>>> > ssl_manager
>>>>>>>>> > 20501 ssl_otp_ca_file_ref set 0 291
>>>>>>>>> > ssl_manager
>>>>>>>>> > 24598 ssl_otp_pem_cache set 3 360
>>>>>>>>> > ssl_manager
>>>>>>>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893
>>>>>>>>> > ssl_manager
>>>>>>>>> > 32797 dets duplicate_bag 2 308
>>>>>>>>> > dets
>>>>>>>>> > 40990 ign_requests set 0 291
>>>>>>>>> > inet_gethost_native
>>>>>>>>> > 45087 ign_req_index set 0 291
>>>>>>>>> > inet_gethost_native
>>>>>>>>> > 2261635104 shell_records ordered_set 0 81
>>>>>>>>> > <0.30638.37>
>>>>>>>>> > ac_tab ac_tab set 33 2216
>>>>>>>>> > application_controller
>>>>>>>>> > code_map code_map set 100 2791 <0.72.0>
>>>>>>>>> > config config set 12 892 <0.72.0>
>>>>>>>>> > dets_owners dets_owners set 1 298 dets
>>>>>>>>> > dets_registry dets_registry set 1 299 dets
>>>>>>>>> > file_io_servers file_io_servers set 1 344
>>>>>>>>> > file_server_2
>>>>>>>>> > global_locks global_locks set 0 291
>>>>>>>>> > global_name_server
>>>>>>>>> > global_names global_names set 0 291
>>>>>>>>> > global_name_server
>>>>>>>>> > global_names_ext global_names_ext set 0 291
>>>>>>>>> > global_name_server
>>>>>>>>> > global_pid_ids global_pid_ids bag 0 291
>>>>>>>>> > global_name_server
>>>>>>>>> > global_pid_names global_pid_names bag 0 291
>>>>>>>>> > global_name_server
>>>>>>>>> > httpc_manager__handler_db httpc_manager__handler_db set 0
>>>>>>>>> > 291 httpc_manager
>>>>>>>>> > httpc_manager__session_db httpc_manager__session_db set 0
>>>>>>>>> > 291 httpc_manager
>>>>>>>>> > inet_cache inet_cache bag 0 291 inet_db
>>>>>>>>> > inet_db inet_db set 29 600 inet_db
>>>>>>>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291
>>>>>>>>> > inet_db
>>>>>>>>> > inet_hosts_byname inet_hosts_byname bag 0 291
>>>>>>>>> > inet_db
>>>>>>>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291
>>>>>>>>> > inet_db
>>>>>>>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291
>>>>>>>>> > inet_db
>>>>>>>>> > models models set 3 28952 <0.72.0>
>>>>>>>>> > sys_dist sys_dist set 1 334
>>>>>>>>> > net_kernel
>>>>>>>>> > ok
>>>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache).
>>>>>>>>> > undefined
>>>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024.
>>>>>>>>> > 65796.0390625
>>>>>>>>> > (n1@REDACTED)8> memory().
>>>>>>>>> > [{total,92699464},
>>>>>>>>> > {processes,8964000},
>>>>>>>>> > {processes_used,8963152},
>>>>>>>>> > {system,83735464},
>>>>>>>>> > {atom,429569},
>>>>>>>>> > {atom_used,421768},
>>>>>>>>> > {binary,199040},
>>>>>>>>> > {code,10411520},
>>>>>>>>> > {ets,69163032}]
>>>>>>>>> >
>>>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly
>>>>>>>>> > 64 MB in 12 hours.
>>>>>>>>> >
>>>>>>>>> > On an OpenBSD platform such process gets killed immediately it
>>>>>>>>> > hits some memory and/or CPU limits.
>>>>>>>>> >
>>>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory,
>>>>>>>>> > otherwise the Erlang VM would be killed.
>>>>>>>>> >
>>>>>>>>> > How can one control , tweak or configure this table such that it
>>>>>>>>> > does not accumulate such data at such high rate.
>>>>>>>>> >
>>>>>>>>> > I seems the table being created private, and there is no way to
>>>>>>>>> > ets:delete_all_objects/1 from table manually.
>>>>>>>>> >
>>>>>>>>> > I know that this table caches some SSL data related to clients,
>>>>>>>>> > but the client has the same IP address,
>>>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection
>>>>>>>>> > info about the same client when only the ephemeral peer port
>>>>>>>>> > differs?
>>>>>>>>> >
>>>>>>>>> > How the size of this table can be held in reasonable limits and
>>>>>>>>> > the rate it's size increases ?
>>>>>>>>> >
>>>>>>>>> > Please if somebody shed some light on these issues.
>>>>>>>>> >
>>>>>>>>> > Thank you,
>>>>>>>>> >
>>>>>>>>> > Bogdan
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > erlang-questions mailing list
>>>>>>>>> > erlang-questions@REDACTED
>>>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>>
>>>>>>>>> "Installing applications can lead to corruption over time.
>>>>>>>>> Applications gradually write over each other's libraries, partial upgrades
>>>>>>>>> occur, user and system errors happen, and minute changes may be unnoticeable
>>>>>>>>> and difficult to fix"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> erlang-questions mailing list
>>>>>>>>> erlang-questions@REDACTED
>>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
More information about the erlang-questions
mailing list