[erlang-questions] 'ssl_otp_session_cache' ets table
Ingela Andin
ingela.andin@REDACTED
Thu Nov 6 11:28:35 CET 2014
Hi!
2014-11-06 8:53 GMT+01:00 Bogdan Andu <bog495@REDACTED>:
> Hi,
>
> on my production servers I have relayd (on OpenBSD) daemon as a reverse
> proxy to some webservers
> where one can fine tune some connection parameters, as well as some ssl
> parameters.
>
> I give a snippet from a relayd.conf configuration file on one of my
> production server:
>
> .............
> #
> # Relay and protocol for HTTP layer 7 loadbalancing and SSL acceleration
> #
> http protocol www_ssl_prot {
> header append "$REMOTE_ADDR" to "X-Forwarded-For"
> header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
> header change "Connection" to "close"
>
> response header change "Server" to "Apache 0.1"
>
> # Various TCP performance options
> tcp { nodelay, sack, socket buffer 65536, backlog 128 }
>
> ssl { no sslv2, no sslv3, tlsv1, ciphers "HIGH" }
> ssl session cache disable
> }
>
> the last directive tells relayd not to use ssl cache.
>
> This configuration is working for years and relayd was restarted once by
> accident - my fault.
>
> SO, y question is:
>
> can we have this configurable in Erlang, in other words, we might be able
> to start an erlang vm such as:
>
> erl -ssl session_cache 'disable' -name x@REDACTED ....
>
> The ssl option session_cache can be set to disabled by default and can
> take values either disable or enabled.
>
>
>
You can already disable the reuse of the sessions using the server
option *{reuse_sessions,
boolean()}* which default to true.
The thing we plan to do is to have a configurable limit on the table size
when sessions are reused.
Regards Ingela Erlang/OTP team Ericsson AB
> Just to let you know...
>
> The statistics of a node running from yesterday when I opened this thread
> of discussion, using a single client:
>
> Node: 'n1@REDACTED' (Connected) (17/6.2) unix (openbsd 5.4.0) CPU:2 SMP
> +A:10
> Time: local time 09:48:48, up for 000:20:27:49, 0ms latency,
> Processes: total 681 (RQ 0) at 610275 RpI using 11526.4k (11805.8k
> allocated)
> Memory: Sys 94835.5k, Atom 407.7k/419.5k, Bin 176.4k, Code 9934.7k, Ets
> 80739.2k
>
> ...........
>
> So, can be this made configurable?
>
>
>
> On Wed, Nov 5, 2014 at 8:52 PM, Ingela Andin <ingela.andin@REDACTED>
> wrote:
>
>> Hi!
>>
>> 2014-11-05 14:15 GMT+01:00 Tony Rogvall <tony@REDACTED>:
>>
>>> I think the ssl session times is the problem here, and the lack of a
>>> maximum size.
>>>
>>> You can change the session time in the ssl environment: session_lifetime
>>> The default is set to 24 hours (in seconds) (if I read it correctly, in
>>> ssl_manager.erl)
>>>
>>
>> Default values are always hard. It is the maximum recommended time for a
>> session to live
>> according to the spec.
>>
>>
>>> I guess that a session_cache_size could be a nice thing to have,
>>> limiting the growth of the session cache.
>>>
>>>
>> Yes I agree.
>>
>>
>>> In other words you have to estimate the life time of your clients and
>>> try to find a reasonable session_lifetime to match that, without blowing
>>> up
>>> the system.
>>>
>>> Maybe the ssl_session_cache_api could be used to implement a strategy
>>> with a max size
>>> cache. Retire session least recently used, while performing the update?
>>> There is a time_stamp in the session that that could be used for this
>>> purpose.
>>>
>>> OTP: Why is a fixed limit not implemented in the standard
>>> ssl_session_cache?
>>> Could this be a target for DOS attacks?
>>>
>>
>>
>> We are aware of the problem, and it is on our todo list. One reason it
>> has not had top priority is that on the server side there are often other
>> mechanisms
>> like firewalls and webbserver settings that limits the problem. And the
>> reason why it was not implemented in the first place is that
>> implementations by nature
>> are iterative and at first you are faced with a lot of bigger problems to
>> solve and then you need to iterate and fine tune and fix things that you
>> now have a better understanding of.
>>
>> In current master there is a change to the session table that limits the
>> growth on the client side if the client behaves inappropriate. It also
>> splits the session table into
>> a server and a client table which is a better implementation as the same
>> Erlang node can be both a client and a server at the same time. So if
>> someone feels like contributing a max limit please base it on the master
>> branch, otherwise I suspect someone compiling about it did raise the
>> priority level a little.
>>
>>
>> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>>
>>
>>> /Tony
>>>
>>> > On 5 nov 2014, at 13:08, Bogdan Andu <bog495@REDACTED> wrote:
>>> >
>>> >
>>> >
>>> > Hi,
>>> >
>>> > I performed a series of test regarding the an Erlang SSL server.
>>> >
>>> > In this setup a major role is played by the table called
>>> 'ssl_otp_session_cache', and of course the processes using it.
>>> >
>>> > The problem is that the size of table increases constantly and,
>>> because an ets table does not automatically deallocate memory unless the
>>> object are deleted from that table, the size of table remains the same even
>>> if there no ssl connections to server.
>>> >
>>> > For example, with a single client running 'ad infinitum' the table
>>> increases at a rate of 5 MBytes/hour. In 12 hours there are allocate around
>>> 60 MB of memory only for this table.
>>> >
>>> > Some info about this:
>>> > $ erl -sname console@REDACTED -remsh n1@REDACTED -setcookie operator
>>> > Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:2:2]
>>> [async-threads:10] [kernel-poll:false]
>>> >
>>> > Eshell V6.2 (abort with ^G)
>>> >
>>> >
>>> > (n1@REDACTED)1> ets:i().
>>> > id name type size mem owner
>>> >
>>> ----------------------------------------------------------------------------
>>> > 12 cookies set 0 291 auth
>>> > 4111 code set 410 26132 code_server
>>> > 8208 code_names set 58 7459 code_server
>>> > 12307 httpc_manager__session_cookie_db bag 0 291
>>> httpc_manager
>>> > 16404 ssl_otp_cacertificate_db set 0 291
>>> ssl_manager
>>> > 20501 ssl_otp_ca_file_ref set 0 291 ssl_manager
>>> > 24598 ssl_otp_pem_cache set 3 360 ssl_manager
>>> > 28695 ssl_otp_session_cache ordered_set 138057 8421893
>>> ssl_manager
>>> > 32797 dets duplicate_bag 2 308 dets
>>> > 40990 ign_requests set 0 291
>>> inet_gethost_native
>>> > 45087 ign_req_index set 0 291
>>> inet_gethost_native
>>> > 2261635104 shell_records ordered_set 0 81
>>> <0.30638.37>
>>> > ac_tab ac_tab set 33 2216
>>> application_controller
>>> > code_map code_map set 100 2791 <0.72.0>
>>> > config config set 12 892 <0.72.0>
>>> > dets_owners dets_owners set 1 298 dets
>>> > dets_registry dets_registry set 1 299 dets
>>> > file_io_servers file_io_servers set 1 344 file_server_2
>>> > global_locks global_locks set 0 291
>>> global_name_server
>>> > global_names global_names set 0 291
>>> global_name_server
>>> > global_names_ext global_names_ext set 0 291
>>> global_name_server
>>> > global_pid_ids global_pid_ids bag 0 291
>>> global_name_server
>>> > global_pid_names global_pid_names bag 0 291
>>> global_name_server
>>> > httpc_manager__handler_db httpc_manager__handler_db set 0 291
>>> httpc_manager
>>> > httpc_manager__session_db httpc_manager__session_db set 0 291
>>> httpc_manager
>>> > inet_cache inet_cache bag 0 291 inet_db
>>> > inet_db inet_db set 29 600 inet_db
>>> > inet_hosts_byaddr inet_hosts_byaddr bag 0 291 inet_db
>>> > inet_hosts_byname inet_hosts_byname bag 0 291 inet_db
>>> > inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 291
>>> inet_db
>>> > inet_hosts_file_byname inet_hosts_file_byname bag 0 291
>>> inet_db
>>> > models models set 3 28952 <0.72.0>
>>> > sys_dist sys_dist set 1 334 net_kernel
>>> > ok
>>> > (n1@REDACTED)5> ets:info(ssl_otp_session_cache).
>>> > undefined
>>> > (n1@REDACTED)7> (8421893*8)/1024.
>>> > 65796.0390625
>>> > (n1@REDACTED)8> memory().
>>> > [{total,92699464},
>>> > {processes,8964000},
>>> > {processes_used,8963152},
>>> > {system,83735464},
>>> > {atom,429569},
>>> > {atom_used,421768},
>>> > {binary,199040},
>>> > {code,10411520},
>>> > {ets,69163032}]
>>> >
>>> > The memory allocated to table 'ssl_otp_session_cache' is roughly 64 MB
>>> in 12 hours.
>>> >
>>> > On an OpenBSD platform such process gets killed immediately it hits
>>> some memory and/or CPU limits.
>>> >
>>> > To make this test on OpenBSD I had to put 'infinit' to memory,
>>> otherwise the Erlang VM would be killed.
>>> >
>>> > How can one control , tweak or configure this table such that it does
>>> not accumulate such data at such high rate.
>>> >
>>> > I seems the table being created private, and there is no way to
>>> ets:delete_all_objects/1 from table manually.
>>> >
>>> > I know that this table caches some SSL data related to clients, but
>>> the client has the same IP address,
>>> > and I wonder why is neccesary to store a lot of SSL connection info
>>> about the same client when only the ephemeral peer port
>>> > differs?
>>> >
>>> > How the size of this table can be held in reasonable limits and the
>>> rate it's size increases ?
>>> >
>>> > Please if somebody shed some light on these issues.
>>> >
>>> > Thank you,
>>> >
>>> > Bogdan
>>> >
>>> >
>>> > _______________________________________________
>>> > erlang-questions mailing list
>>> > erlang-questions@REDACTED
>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>> "Installing applications can lead to corruption over time. Applications
>>> gradually write over each other's libraries, partial upgrades occur, user
>>> and system errors happen, and minute changes may be unnoticeable and
>>> difficult to fix"
>>>
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141106/92a42edb/attachment.htm>
More information about the erlang-questions
mailing list