snit (SNI Termination Library) to replace Nginx

[Dave Cottlehuber]

On Sat, 9 Nov 2019, at 07:22, Frank Muller wrote:
> We mainly upload large files (20mB to 100mB) to our two webapps behind Nginx.
> ssl_prefer_server_ciphers on;
> ssl_ecdh_curve
> secp384r1

TLDR:

- use TLS1.3 if you can - most of the decisions have been made for you
- ensure your cipher choice is hardware accelerated in your openssl
- look at actual network traffic to see if there's any issues there
- no easy answers, benchmark your setup

I hope this helps point you in the right direction.

"SSL/TLS accounts for less than 1% of the CPU load, less than 10 KB of
memory per connection and less than 2% of network overhead."

-- https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html

It should be possible to transfer traffic over TLS at rates significantly
faster than what you're reporting, obviously. However, I would be surprised
if nginx itself is the problem, given netflix can saturate their pipes with
nginx, admittedly with a lot of tweaking [1], [2] and a custom FreeBSD build.

I would first look to see if you can restrict your ciphers to provide better
performance for your hardware, and highly recommend capturing data with
tcpdump & wireshark to do some network level analysis. This will vary a lot
depending on what control you have over client TLS capabilities [3], and
if you have OpenSSL 1.1.x available, and perhaps http2 on clients also.

Intel's notes from 2016 [4] show a noticeable difference between algorithms
so you need to benchmark your load on your hardware.

Personally, for TLS termination I prefer haproxy[5] but all of hitch, nginx,
snit, haproxy should be able to achieve similar results.[6] is interesting
but 2.x haproxy handles multiple processes itself.

You can use https://www.ssllabs.com/ssltest/analyze.html or
https://github.com/drwetter/testssl.sh to help validate protocol choices.

Useful references:

- https://istlsfastyet.com/
- https://hpbn.co/transport-layer-security-tls/
- https://www.haproxy.com/knowledge-base/ssl/
- https://www.feistyduck.com/books/bulletproof-ssl-and-tls/ get the ebook
direct as amazon seems to have an out of date version

[0]: https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html
[1]: https://openconnect.netflix.com/publications/asiabsd_2015_tls.pdf
[2]: https://openconnect.netflix.com/publications/asiabsd_tls_improved.pdf
[3]: https://wiki.mozilla.org/Security/Server_Side_TLS#Recommended_Ciphersuite
[4]: https://software.intel.com/en-us/articles/accelerating-ssl-load-balancers-with-intel-xeon-e5-v4-processors
[5]: https://www.haproxy.com/blog/haproxy-ssl-termination/
[6]: https://www.freecodecamp.org/news/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27/
[7]: https://www.ssllabs.com/ssltest/analyze.html
[8]: https://github.com/drwetter/testssl.sh