[erlang-questions] 'ssl_otp_session_cache' ets table

Bogdan Andu bog495@REDACTED
Thu Nov 6 17:20:50 CET 2014


yes,

in function ssl_manager:invalidate_session/4, the call

erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}), does
the actual deletion of session from cache

where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which is
60 seconds




On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin <ingela.andin@REDACTED> wrote:

> Hi!
>
>
> 2014-11-06 14:39 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>
>> Hi,
>>
>> i have tried to pass to execute th following forms of commands:
>>
>> erl  -ssl session_lifetime 60
>>
>> erl  -ssl session_lifetime '60'
>>
>> erl  -ssl session_lifetime '[60]'
>>
>>
>> in order to set a ssl session_lifetime of 60 seconds.
>>
>> in init of ssl_manager , after session_lifetime + 5 seconds a process is
>> created to validate or invalidate sessions.
>> So, after 65 seconds a foldl is called on sessions table to check
>> sessions after formula:
>> Now - TimeStamp < LifeTime
>>
>> However, even after I stop the client, the table 'ssl_otp_session_cache'
>> 's size remains the same.
>>
>>
>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the table
>> is sweeped to delete any expired sessions,
>>
>> but also nothing happens.
>>
>>
> There is also a delay in the actual deletion. In the first sweep the
> sessions will only be invalidated as there may already be
> spawned connection handlers that needs to read the session data before we
> may delete it. It is a performance trade off.
>
> Regards Ingela Erlang/OTP team - Ericsson AB
>
>
>
>>
>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin <ingela.andin@REDACTED>
>> wrote:
>>
>>> Hi!
>>>
>>>
>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>
>>>> Hi,
>>>>
>>>> on my production servers I have relayd (on OpenBSD) daemon as a reverse
>>>> proxy to some webservers
>>>> where one can fine tune some connection parameters, as well as some ssl
>>>> parameters.
>>>>
>>>> I give a snippet from a relayd.conf configuration file on one of my
>>>> production server:
>>>>
>>>> .............
>>>> #
>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration
>>>> #
>>>> http protocol www_ssl_prot {
>>>>         header append "$REMOTE_ADDR" to "X-Forwarded-For"
>>>>         header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
>>>>         header change "Connection" to "close"
>>>>
>>>>         response header change "Server" to "Apache 0.1"
>>>>
>>>>         # Various TCP performance options
>>>>         tcp { nodelay, sack, socket buffer 65536, backlog 128 }
>>>>
>>>>         ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" }
>>>>         ssl session cache disable
>>>> }
>>>>
>>>> the last directive tells relayd not to use ssl cache.
>>>>
>>>> This configuration is working for years and relayd was restarted once
>>>> by accident - my fault.
>>>>
>>>> SO, y question is:
>>>>
>>>> can we have this configurable in Erlang, in other words, we might be
>>>> able to start an erlang vm such as:
>>>>
>>>> erl -ssl session_cache 'disable' -name x@REDACTED ....
>>>>
>>>> The ssl option session_cache can be set to disabled by default and can
>>>> take values either disable or enabled.
>>>>
>>>>
>>>>
>>> You can already disable the reuse of the sessions using the server
>>> option *{reuse_sessions, boolean()}* which default to true.
>>> The thing we plan to do is to have a configurable limit on the table
>>> size when sessions are reused.
>>>
>>> Regards Ingela Erlang/OTP team Ericsson AB
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Just to let you know...
>>>>
>>>> The statistics of a node running from yesterday when  I opened this
>>>> thread of discussion, using a single client:
>>>>
>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2
>>>> SMP +A:10
>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency,
>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k
>>>> allocated)
>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets
>>>> 80739.2k
>>>>
>>>> ...........
>>>>
>>>> So, can be this made configurable?
>>>>
>>>>
>>>>
>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin <ingela.andin@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall <tony@REDACTED>:
>>>>>
>>>>>> I think the ssl session times is the problem here, and the lack of a
>>>>>> maximum size.
>>>>>>
>>>>>> You can change the session time in the ssl environment:
>>>>>> session_lifetime
>>>>>> The default is set to 24 hours (in seconds) (if I read it correctly,
>>>>>> in ssl_manager.erl)
>>>>>>
>>>>>
>>>>> Default values are always hard. It is the maximum recommended time for
>>>>> a session to live
>>>>> according to the spec.
>>>>>
>>>>>
>>>>>> I guess that a session_cache_size could be a nice thing to have,
>>>>>> limiting the growth of the session cache.
>>>>>>
>>>>>>
>>>>> Yes I agree.
>>>>>
>>>>>
>>>>>> In other words you have to estimate the life time of your clients and
>>>>>> try to find a reasonable session_lifetime to match that, without
>>>>>> blowing up
>>>>>> the system.
>>>>>>
>>>>>> Maybe the ssl_session_cache_api could be used to implement a strategy
>>>>>> with a max size
>>>>>> cache. Retire session least recently used, while performing the
>>>>>> update?
>>>>>> There is a time_stamp in the session that that could be used for this
>>>>>> purpose.
>>>>>>
>>>>>> OTP: Why is a fixed limit not implemented in the standard
>>>>>> ssl_session_cache?
>>>>>> Could this be a target for DOS attacks?
>>>>>>
>>>>>
>>>>>
>>>>> We are aware of the problem, and it is on our todo list. One reason it
>>>>> has not had top priority is that on the server side there are often other
>>>>> mechanisms
>>>>> like firewalls and webbserver settings that limits the problem.  And
>>>>> the reason why it was not implemented in the first place is that
>>>>> implementations by nature
>>>>> are iterative and at first you are faced with a lot of bigger problems
>>>>> to solve and then you need to iterate and fine tune and fix things that you
>>>>> now have a better understanding of.
>>>>>
>>>>> In current master there is a change to the session table that limits
>>>>> the growth on the client side if the client behaves inappropriate. It also
>>>>> splits the session table into
>>>>> a server and a client table which is a better implementation as the
>>>>> same Erlang node can be both a client and a server at the same time. So if
>>>>>  someone feels like contributing a max limit please base it on the master
>>>>>  branch, otherwise I suspect someone compiling about it did raise the
>>>>> priority level  a little.
>>>>>
>>>>>
>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>
>>>>>
>>>>>
>>>>>> /Tony
>>>>>>
>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu <bog495@REDACTED> wrote:
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Hi,
>>>>>> >
>>>>>> > I performed a series of test regarding the an Erlang SSL server.
>>>>>> >
>>>>>> > In this setup a major role is played by the table called
>>>>>> 'ssl_otp_session_cache', and of course the processes using it.
>>>>>> >
>>>>>> > The problem is that the size of table increases constantly and,
>>>>>> because an ets table does not automatically deallocate memory unless the
>>>>>> object are deleted from that table, the size of table remains the same even
>>>>>> if there no ssl connections to server.
>>>>>> >
>>>>>> > For example, with a single client running 'ad infinitum' the table
>>>>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around
>>>>>> 60 MB of memory only for this table.
>>>>>> >
>>>>>> > Some info about this:
>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie
>>>>>> operator
>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2]
>>>>>> [async-threads:10] [kernel-poll:false]
>>>>>> >
>>>>>> > Eshell V6.2  (abort with ^G)
>>>>>> >
>>>>>> >
>>>>>> > (n1@REDACTED)1> ets:i().
>>>>>> >  id              name              type  size   mem      owner
>>>>>> >
>>>>>> ----------------------------------------------------------------------------
>>>>>> >  12              cookies           set   0      291      auth
>>>>>> >  4111            code              set   410    26132    code_server
>>>>>> >  8208            code_names        set   58     7459     code_server
>>>>>> >  12307           httpc_manager__session_cookie_db bag   0      291
>>>>>>     httpc_manager
>>>>>> >  16404           ssl_otp_cacertificate_db set   0      291
>>>>>> ssl_manager
>>>>>> >  20501           ssl_otp_ca_file_ref set   0      291
>>>>>> ssl_manager
>>>>>> >  24598           ssl_otp_pem_cache set   3      360      ssl_manager
>>>>>> >  28695           ssl_otp_session_cache ordered_set 138057 8421893
>>>>>> ssl_manager
>>>>>> >  32797           dets              duplicate_bag 2      308
>>>>>> dets
>>>>>> >  40990           ign_requests      set   0      291
>>>>>> inet_gethost_native
>>>>>> >  45087           ign_req_index     set   0      291
>>>>>> inet_gethost_native
>>>>>> >  2261635104      shell_records     ordered_set 0      81
>>>>>>  <0.30638.37>
>>>>>> >  ac_tab          ac_tab            set   33     2216
>>>>>>  application_controller
>>>>>> >  code_map        code_map          set   100    2791     <0.72.0>
>>>>>> >  config          config            set   12     892      <0.72.0>
>>>>>> >  dets_owners     dets_owners       set   1      298      dets
>>>>>> >  dets_registry   dets_registry     set   1      299      dets
>>>>>> >  file_io_servers file_io_servers   set   1      344
>>>>>> file_server_2
>>>>>> >  global_locks    global_locks      set   0      291
>>>>>> global_name_server
>>>>>> >  global_names    global_names      set   0      291
>>>>>> global_name_server
>>>>>> >  global_names_ext global_names_ext  set   0      291
>>>>>> global_name_server
>>>>>> >  global_pid_ids  global_pid_ids    bag   0      291
>>>>>> global_name_server
>>>>>> >  global_pid_names global_pid_names  bag   0      291
>>>>>> global_name_server
>>>>>> >  httpc_manager__handler_db httpc_manager__handler_db set   0
>>>>>> 291      httpc_manager
>>>>>> >  httpc_manager__session_db httpc_manager__session_db set   0
>>>>>> 291      httpc_manager
>>>>>> >  inet_cache      inet_cache        bag   0      291      inet_db
>>>>>> >  inet_db         inet_db           set   29     600      inet_db
>>>>>> >  inet_hosts_byaddr inet_hosts_byaddr bag   0      291      inet_db
>>>>>> >  inet_hosts_byname inet_hosts_byname bag   0      291      inet_db
>>>>>> >  inet_hosts_file_byaddr inet_hosts_file_byaddr bag   0      291
>>>>>>   inet_db
>>>>>> >  inet_hosts_file_byname inet_hosts_file_byname bag   0      291
>>>>>>   inet_db
>>>>>> >  models          models            set   3      28952    <0.72.0>
>>>>>> >  sys_dist        sys_dist          set   1      334      net_kernel
>>>>>> > ok
>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache).
>>>>>> > undefined
>>>>>> > (n1@REDACTED)7> (8421893*8)/1024.
>>>>>> > 65796.0390625
>>>>>> > (n1@REDACTED)8> memory().
>>>>>> > [{total,92699464},
>>>>>> >  {processes,8964000},
>>>>>> >  {processes_used,8963152},
>>>>>> >  {system,83735464},
>>>>>> >  {atom,429569},
>>>>>> >  {atom_used,421768},
>>>>>> >  {binary,199040},
>>>>>> >  {code,10411520},
>>>>>> >  {ets,69163032}]
>>>>>> >
>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64
>>>>>> MB in 12 hours.
>>>>>> >
>>>>>> > On an OpenBSD platform such process gets killed immediately it hits
>>>>>> some memory and/or CPU limits.
>>>>>> >
>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory,
>>>>>> otherwise the Erlang VM would be killed.
>>>>>> >
>>>>>> > How can one control , tweak or configure this table such that it
>>>>>> does not accumulate such data at such high rate.
>>>>>> >
>>>>>> > I seems the table being created private, and there is no way to
>>>>>> ets:delete_all_objects/1 from table manually.
>>>>>> >
>>>>>> > I know that this table caches some SSL data related to clients, but
>>>>>> the client has the same IP address,
>>>>>> > and I wonder why is neccesary to store a lot of SSL connection info
>>>>>> about the same client when only the ephemeral peer port
>>>>>> > differs?
>>>>>> >
>>>>>> > How the size of this table can be held in reasonable limits and the
>>>>>> rate it's size increases ?
>>>>>> >
>>>>>> > Please if somebody shed some light on these issues.
>>>>>> >
>>>>>> > Thank you,
>>>>>> >
>>>>>> > Bogdan
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > erlang-questions mailing list
>>>>>> > erlang-questions@REDACTED
>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>
>>>>>> "Installing applications can lead to corruption over time.
>>>>>> Applications gradually write over each other's libraries, partial upgrades
>>>>>> occur, user and system errors happen, and minute changes may be
>>>>>> unnoticeable and difficult to fix"
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> erlang-questions mailing list
>>>>>> erlang-questions@REDACTED
>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141106/eb9e394e/attachment.htm>


More information about the erlang-questions mailing list