snit (SNI Termination Library) to replace Nginx

[Frank Muller]

Hi Mikael

We mainly upload large files (20mB to 100mB) to our two webapps behind
Nginx.

And yes, we tried these 2 options but they dont help in this situation.

In our case, we are completely disabling buffering on Nginx (a feature
introduced in version Nginx 1.7.3 taken from its fork Tengine at Taobao:
https://tengine.taobao.org). Disabling buffer was a big win, but still much
slower than direct HTTP access via cowboy.

How did we find out that Nginx was the culprit?
Simply by testing with another TLS termination proxy called Hitch (from
Varnish):
https://hitch-tls.org

Hitch is 1.5x - 2x slower than Cowboy. Unfortunately it only supports one
upstream backend server at a time. Thus, we can’t serve our two webapps on
port 443. Another constraint is that our two webapps has to run on the same
host (a customer’s requirement).

Finally the system is not even under load. Maximum of 10 files upload per
hour.

Forgot to mention our config:

1. Erlang 22.1.6
2. Linux kernel 4.15.0.66 / Ubuntu LTS 18.10 x86_64
3. Physical machine: 32gB of RAM, 8-Cores Intel Xeon CPU E3-1270 v6@REDACTED
4. Nginx V1.14.0
5. Sysctl tuned by our engineers for handle fast TCP connections with
enough open files limits (ulimit -n: 200000)

/Frank

Le sam. 9 nov. 2019 à 03:58, Mikael Karlsson <karlsson.rm@REDACTED> a
écrit :

> Hi,
> Did you try with proxy_buffering set to on, and/or changing the
> proxy_buffer_size?
> Regards Mikael
>
>
> Den lör 9 nov. 2019 00:14Frank Muller <frank.muller.erl@REDACTED> skrev:
>
>> Hi guys
>>
>> Anyone familiar with snit?
>> https://github.com/heroku/snit
>>
>> We’re facing a performance issue with Nginx used as TLS Termination.
>> Nginx is in front of our two Erlang webapps. Both running on the same
>> machine, and both based on Cowboy 2.7.0.
>>
>> The problem:
>> [1] directly accessing the two webapps (plain HTTP) is fast enough for
>> us, and Cowboy is doing just great.
>> [2] accessing any of the two apps with Nginx (HTTPS) is 3x-5x slower
>> than  in [1]
>>
>> We selected Nginx for its ability to hide our apps, and be able to access
>> them both on port 443 (default HTTPS).
>>
>> Our Nginx config is pretty simple, tuned for SSL/TLS.
>> ______________________________________________
>> server {
>>         listen  443 ssl;
>>
>>         server_name  app1.acme.com; # the 2nd webapp is running on:
>> app2.acme.com
>>
>>         ssl on;
>>         ssl_certificate         /etc/nginx/certs/app1/crt.pem;
>>         ssl_certificate_key /etc/nginx/certs/app1/key.pem;
>>         ssl_dhparam         /etc/nginx/certs/app1/dh.pem;
>>
>>         ssl_protocols       TLSv1.2;
>>
>>         ssl_prefer_server_ciphers on;
>>
>>         ssl_ecdh_curve
>>         secp384r1
>>
>>         ssl_session_cache shared:SSL:50m;
>>         ssl_session_timeout  1d;
>>         ssl_session_tickets off;
>>
>>         ssl_stapling on;
>>         ssl_stapling_verify on;
>>
>>         resolver 8.8.8.8 8.8.4.4 valid=300s;
>>         resolver_timeout 5s;
>>
>>         ssl_buffer_size 8k;
>>
>>         keepalive_timeout 0;
>>
>>
>>         client_max_body_size 0;
>>         client_body_buffer_size 4m;
>>         client_header_timeout  300;
>>         client_body_timeout    300;
>>         client_header_buffer_size    1k;
>>         large_client_header_buffers  4 4k;
>>
>>         location = /favicon.ico {
>>            access_log off;
>>            return 204;
>>         }
>>
>>         location / {
>>            send_timeout           5;
>>
>>            proxy_http_version 1.1;
>>            proxy_buffering off;
>>            proxy_request_buffering off;
>>            proxy_ignore_headers "Cache-Control" "Expires";
>>            proxy_max_temp_file_size 30m;
>>            proxy_connect_timeout 300;
>>            proxy_read_timeout 300;
>>            proxy_send_timeout 300;
>>            proxy_intercept_errors off;
>>
>>            proxy_set_header        X-Real-IP       $remote_addr;
>>            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>
>>            proxy_pass http://127.0.0.1:2222; # the 2nd webapp has:
>> proxy_pass http://127.0.0.1:3333;
>>         }
>> }
>> ______________________________________________
>>
>> Can snit be used to replace Nginx?
>> Help and suggestions appreciated.
>>
>> Best
>> /Frank
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20191109/3de694df/attachment.htm>