[erlang-questions] 'ssl_otp_session_cache' ets table

Bogdan Andu bog495@REDACTED
Fri Nov 7 11:58:59 CET 2014


I work with R14B04 as production and OTP 17.3 as experimental.

R14B04 also deletes all entries after 24 hours.

17.3 also deletes all entries after 24 hours.

I think is it has to do with passing the environment parameter like this:

erl  -ssl session_lifetime 60 ...

I tried many variants and may be I miss something.

may be I am mistaken but it is possible that only the default value, 24
hours,
is only considered as the environment variable is not passed correctly by
me.

i think the environment parameters are the problem here because maybe  i
specify them incorrctly.

could you advise what is the proper method of correctly specifying
session_lifetime, because according to
manual is like :

erl  -ssl session_lifetime 60 ...

a ssl session must expire afte 60 seconds + delayed 60 seconds = 2 minutes.

intstead entries from 'ssl_otp_session_cache' be deleted every 2 minutes,
they remain 24 hours .

after 22 hours I have:

Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP
+A:10
Time: local time 12:50:23, up for 000:21:54:49, 0ms latency,
Processes: total 676 (RQ 0) at 21121 RpI using 10978.0k (11007.4k allocated)
Memory: Sys 14898.8k, Atom 410.8k/419.5k, Bin 112.9k, Code 10053.7k, Ets
720.6k


and for ets:i().

...........
 28695           ssl_otp_session_cache ordered_set 115    7134
ssl_manager
..........

The vm was configured with 60 seconds session lifetime and the client was
running 60 seconds
an I expected after 2 minutes to see less entries in the table but they
remained untouched


I will see what happens after 24 hours mark and I'll let you know

and another thing...

even if i pass the options:
{reuse_session, fun(_,_,_,_) -> false end},
{reuse_sessions, false},

both or individually , to the ssl:listen/2 function the table is filled
with sessions and they expire after 24 hours,
only that cpu is working harder, 7-8% which means the sessions a not reused
and the ssl
handshake is happening every time.


Bogdan

On Fri, Nov 7, 2014 at 10:37 AM, Ingela Andin <ingela.andin@REDACTED>
wrote:

> Hi!
>
> Which version of the ssl application are you using? The session cleanup
> was broken in one of the older versions.
>
> Regards Ingela Erlang/OTP - team Ericsson AB
>
>
> 2014-11-06 17:20 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>
>> yes,
>>
>> in function ssl_manager:invalidate_session/4, the call
>>
>> erlang:send_after(delay_time(), self(), {delayed_clean_session, Key}),
>> does the actual deletion of session from cache
>>
>> where delay_time/0 function returns ?CLEAN_SESSION_DB by default, which
>> is 60 seconds
>>
>>
>>
>>
>> On Thu, Nov 6, 2014 at 4:51 PM, Ingela Andin <ingela.andin@REDACTED>
>> wrote:
>>
>>> Hi!
>>>
>>>
>>> 2014-11-06 14:39 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>
>>>> Hi,
>>>>
>>>> i have tried to pass to execute th following forms of commands:
>>>>
>>>> erl  -ssl session_lifetime 60
>>>>
>>>> erl  -ssl session_lifetime '60'
>>>>
>>>> erl  -ssl session_lifetime '[60]'
>>>>
>>>>
>>>> in order to set a ssl session_lifetime of 60 seconds.
>>>>
>>>> in init of ssl_manager , after session_lifetime + 5 seconds a process
>>>> is created to validate or invalidate sessions.
>>>> So, after 65 seconds a foldl is called on sessions table to check
>>>> sessions after formula:
>>>> Now - TimeStamp < LifeTime
>>>>
>>>> However, even after I stop the client, the table
>>>> 'ssl_otp_session_cache' 's size remains the same.
>>>>
>>>>
>>>> After that once every ?CLEAN_SESSION_DB which is 60 seconds , the
>>>> table is sweeped to delete any expired sessions,
>>>>
>>>> but also nothing happens.
>>>>
>>>>
>>> There is also a delay in the actual deletion. In the first sweep the
>>> sessions will only be invalidated as there may already be
>>> spawned connection handlers that needs to read the session data before
>>> we may delete it. It is a performance trade off.
>>>
>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>
>>>
>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 12:28 PM, Ingela Andin <ingela.andin@REDACTED>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>>
>>>>> 2014-11-06 8:53 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> on my production servers I have relayd (on OpenBSD) daemon as a
>>>>>> reverse proxy to some webservers
>>>>>> where one can fine tune some connection parameters, as well as some
>>>>>> ssl parameters.
>>>>>>
>>>>>> I give a snippet from a relayd.conf configuration file on one of my
>>>>>> production server:
>>>>>>
>>>>>> .............
>>>>>> #
>>>>>> # Relay and protocol for HTTP layer 7 loadbalancing and SSL
>>>>>> acceleration
>>>>>> #
>>>>>> http protocol www_ssl_prot {
>>>>>>         header append "$REMOTE_ADDR" to "X-Forwarded-For"
>>>>>>         header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
>>>>>>         header change "Connection" to "close"
>>>>>>
>>>>>>         response header change "Server" to "Apache 0.1"
>>>>>>
>>>>>>         # Various TCP performance options
>>>>>>         tcp { nodelay, sack, socket buffer 65536, backlog 128 }
>>>>>>
>>>>>>         ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" }
>>>>>>         ssl session cache disable
>>>>>> }
>>>>>>
>>>>>> the last directive tells relayd not to use ssl cache.
>>>>>>
>>>>>> This configuration is working for years and relayd was restarted once
>>>>>> by accident - my fault.
>>>>>>
>>>>>> SO, y question is:
>>>>>>
>>>>>> can we have this configurable in Erlang, in other words, we might be
>>>>>> able to start an erlang vm such as:
>>>>>>
>>>>>> erl -ssl session_cache 'disable' -name x@REDACTED ....
>>>>>>
>>>>>> The ssl option session_cache can be set to disabled by default and
>>>>>> can take values either disable or enabled.
>>>>>>
>>>>>>
>>>>>>
>>>>> You can already disable the reuse of the sessions using the server
>>>>> option *{reuse_sessions, boolean()}* which default to true.
>>>>> The thing we plan to do is to have a configurable limit on the table
>>>>> size when sessions are reused.
>>>>>
>>>>> Regards Ingela Erlang/OTP team Ericsson AB
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Just to let you know...
>>>>>>
>>>>>> The statistics of a node running from yesterday when  I opened this
>>>>>> thread of discussion, using a single client:
>>>>>>
>>>>>> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2
>>>>>> SMP +A:10
>>>>>> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency,
>>>>>> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k
>>>>>> allocated)
>>>>>> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k,
>>>>>> Ets 80739.2k
>>>>>>
>>>>>> ...........
>>>>>>
>>>>>> So, can be this made configurable?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin <ingela.andin@REDACTED>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall <tony@REDACTED>:
>>>>>>>
>>>>>>>> I think the ssl session times is the problem here, and the lack of
>>>>>>>> a maximum size.
>>>>>>>>
>>>>>>>> You can change the session time in the ssl environment:
>>>>>>>> session_lifetime
>>>>>>>> The default is set to 24 hours (in seconds) (if I read it
>>>>>>>> correctly, in ssl_manager.erl)
>>>>>>>>
>>>>>>>
>>>>>>> Default values are always hard. It is the maximum recommended time
>>>>>>> for a session to live
>>>>>>> according to the spec.
>>>>>>>
>>>>>>>
>>>>>>>> I guess that a session_cache_size could be a nice thing to have,
>>>>>>>> limiting the growth of the session cache.
>>>>>>>>
>>>>>>>>
>>>>>>> Yes I agree.
>>>>>>>
>>>>>>>
>>>>>>>> In other words you have to estimate the life time of your clients
>>>>>>>> and
>>>>>>>> try to find a reasonable session_lifetime to match that, without
>>>>>>>> blowing up
>>>>>>>> the system.
>>>>>>>>
>>>>>>>> Maybe the ssl_session_cache_api could be used to implement a
>>>>>>>> strategy with a max size
>>>>>>>> cache. Retire session least recently used, while performing the
>>>>>>>> update?
>>>>>>>> There is a time_stamp in the session that that could be used for
>>>>>>>> this purpose.
>>>>>>>>
>>>>>>>> OTP: Why is a fixed limit not implemented in the standard
>>>>>>>> ssl_session_cache?
>>>>>>>> Could this be a target for DOS attacks?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We are aware of the problem, and it is on our todo list. One reason
>>>>>>> it has not had top priority is that on the server side there are often
>>>>>>> other mechanisms
>>>>>>> like firewalls and webbserver settings that limits the problem.  And
>>>>>>> the reason why it was not implemented in the first place is that
>>>>>>> implementations by nature
>>>>>>> are iterative and at first you are faced with a lot of bigger
>>>>>>> problems to solve and then you need to iterate and fine tune and fix things
>>>>>>> that you now have a better understanding of.
>>>>>>>
>>>>>>> In current master there is a change to the session table that limits
>>>>>>> the growth on the client side if the client behaves inappropriate. It also
>>>>>>> splits the session table into
>>>>>>> a server and a client table which is a better implementation as the
>>>>>>> same Erlang node can be both a client and a server at the same time. So if
>>>>>>>  someone feels like contributing a max limit please base it on the master
>>>>>>>  branch, otherwise I suspect someone compiling about it did raise the
>>>>>>> priority level  a little.
>>>>>>>
>>>>>>>
>>>>>>> Regards Ingela Erlang/OTP team - Ericsson AB
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> /Tony
>>>>>>>>
>>>>>>>> > On 5 nov 2014, at 13:08, Bogdan Andu <bog495@REDACTED> wrote:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > I performed a series of test regarding the an Erlang SSL server.
>>>>>>>> >
>>>>>>>> > In this setup a major role is played by the table called
>>>>>>>> 'ssl_otp_session_cache', and of course the processes using it.
>>>>>>>> >
>>>>>>>> > The problem is that the size of table increases constantly and,
>>>>>>>> because an ets table does not automatically deallocate memory unless the
>>>>>>>> object are deleted from that table, the size of table remains the same even
>>>>>>>> if there no ssl connections to server.
>>>>>>>> >
>>>>>>>> > For example, with a single client running 'ad infinitum' the
>>>>>>>> table increases at a rate of 5 MBytes/hour. In 12 hours there are allocate
>>>>>>>> around 60 MB of memory only for this table.
>>>>>>>> >
>>>>>>>> > Some info about this:
>>>>>>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie
>>>>>>>> operator
>>>>>>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2]
>>>>>>>> [async-threads:10] [kernel-poll:false]
>>>>>>>> >
>>>>>>>> > Eshell V6.2  (abort with ^G)
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > (n1@REDACTED)1> ets:i().
>>>>>>>> >  id              name              type  size   mem      owner
>>>>>>>> >
>>>>>>>> ----------------------------------------------------------------------------
>>>>>>>> >  12              cookies           set   0      291      auth
>>>>>>>> >  4111            code              set   410    26132
>>>>>>>> code_server
>>>>>>>> >  8208            code_names        set   58     7459
>>>>>>>>  code_server
>>>>>>>> >  12307           httpc_manager__session_cookie_db bag   0
>>>>>>>> 291      httpc_manager
>>>>>>>> >  16404           ssl_otp_cacertificate_db set   0      291
>>>>>>>> ssl_manager
>>>>>>>> >  20501           ssl_otp_ca_file_ref set   0      291
>>>>>>>> ssl_manager
>>>>>>>> >  24598           ssl_otp_pem_cache set   3      360
>>>>>>>> ssl_manager
>>>>>>>> >  28695           ssl_otp_session_cache ordered_set 138057
>>>>>>>> 8421893  ssl_manager
>>>>>>>> >  32797           dets              duplicate_bag 2      308
>>>>>>>> dets
>>>>>>>> >  40990           ign_requests      set   0      291
>>>>>>>> inet_gethost_native
>>>>>>>> >  45087           ign_req_index     set   0      291
>>>>>>>> inet_gethost_native
>>>>>>>> >  2261635104      shell_records     ordered_set 0      81
>>>>>>>>  <0.30638.37>
>>>>>>>> >  ac_tab          ac_tab            set   33     2216
>>>>>>>>  application_controller
>>>>>>>> >  code_map        code_map          set   100    2791     <0.72.0>
>>>>>>>> >  config          config            set   12     892      <0.72.0>
>>>>>>>> >  dets_owners     dets_owners       set   1      298      dets
>>>>>>>> >  dets_registry   dets_registry     set   1      299      dets
>>>>>>>> >  file_io_servers file_io_servers   set   1      344
>>>>>>>> file_server_2
>>>>>>>> >  global_locks    global_locks      set   0      291
>>>>>>>> global_name_server
>>>>>>>> >  global_names    global_names      set   0      291
>>>>>>>> global_name_server
>>>>>>>> >  global_names_ext global_names_ext  set   0      291
>>>>>>>> global_name_server
>>>>>>>> >  global_pid_ids  global_pid_ids    bag   0      291
>>>>>>>> global_name_server
>>>>>>>> >  global_pid_names global_pid_names  bag   0      291
>>>>>>>> global_name_server
>>>>>>>> >  httpc_manager__handler_db httpc_manager__handler_db set   0
>>>>>>>> 291      httpc_manager
>>>>>>>> >  httpc_manager__session_db httpc_manager__session_db set   0
>>>>>>>> 291      httpc_manager
>>>>>>>> >  inet_cache      inet_cache        bag   0      291      inet_db
>>>>>>>> >  inet_db         inet_db           set   29     600      inet_db
>>>>>>>> >  inet_hosts_byaddr inet_hosts_byaddr bag   0      291      inet_db
>>>>>>>> >  inet_hosts_byname inet_hosts_byname bag   0      291      inet_db
>>>>>>>> >  inet_hosts_file_byaddr inet_hosts_file_byaddr bag   0      291
>>>>>>>>     inet_db
>>>>>>>> >  inet_hosts_file_byname inet_hosts_file_byname bag   0      291
>>>>>>>>     inet_db
>>>>>>>> >  models          models            set   3      28952    <0.72.0>
>>>>>>>> >  sys_dist        sys_dist          set   1      334
>>>>>>>> net_kernel
>>>>>>>> > ok
>>>>>>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache).
>>>>>>>> > undefined
>>>>>>>> > (n1@REDACTED)7> (8421893*8)/1024.
>>>>>>>> > 65796.0390625
>>>>>>>> > (n1@REDACTED)8> memory().
>>>>>>>> > [{total,92699464},
>>>>>>>> >  {processes,8964000},
>>>>>>>> >  {processes_used,8963152},
>>>>>>>> >  {system,83735464},
>>>>>>>> >  {atom,429569},
>>>>>>>> >  {atom_used,421768},
>>>>>>>> >  {binary,199040},
>>>>>>>> >  {code,10411520},
>>>>>>>> >  {ets,69163032}]
>>>>>>>> >
>>>>>>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly
>>>>>>>> 64 MB in 12 hours.
>>>>>>>> >
>>>>>>>> > On an OpenBSD platform such process gets killed immediately it
>>>>>>>> hits some memory and/or CPU limits.
>>>>>>>> >
>>>>>>>> > To make this test on OpenBSD I had to put 'infinit' to memory,
>>>>>>>> otherwise the Erlang VM would be killed.
>>>>>>>> >
>>>>>>>> > How can one control , tweak or configure this table such that it
>>>>>>>> does not accumulate such data at such high rate.
>>>>>>>> >
>>>>>>>> > I seems the table being created private, and there is no way to
>>>>>>>> ets:delete_all_objects/1 from table manually.
>>>>>>>> >
>>>>>>>> > I know that this table caches some SSL data related to clients,
>>>>>>>> but the client has the same IP address,
>>>>>>>> > and I wonder why is neccesary to store a lot of SSL connection
>>>>>>>> info about the same client when only the ephemeral peer port
>>>>>>>> > differs?
>>>>>>>> >
>>>>>>>> > How the size of this table can be held in reasonable limits and
>>>>>>>> the rate it's size increases ?
>>>>>>>> >
>>>>>>>> > Please if somebody shed some light on these issues.
>>>>>>>> >
>>>>>>>> > Thank you,
>>>>>>>> >
>>>>>>>> > Bogdan
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > erlang-questions mailing list
>>>>>>>> > erlang-questions@REDACTED
>>>>>>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>
>>>>>>>> "Installing applications can lead to corruption over time.
>>>>>>>> Applications gradually write over each other's libraries, partial upgrades
>>>>>>>> occur, user and system errors happen, and minute changes may be
>>>>>>>> unnoticeable and difficult to fix"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> erlang-questions mailing list
>>>>>>>> erlang-questions@REDACTED
>>>>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141107/c6f8b875/attachment.htm>


More information about the erlang-questions mailing list