<div><div dir="auto">Hi Mikael</div></div><div dir="auto"><br></div><div dir="auto">We mainly upload large files (20mB to 100mB) to our two webapps behind Nginx.</div><div dir="auto"><br></div><div dir="auto">And yes, we tried these 2 options but they dont help in this situation. </div><div dir="auto"><br></div><div dir="auto">In our case, we are completely disabling buffering on Nginx (a feature introduced in version Nginx 1.7.3 taken from its fork Tengine at Taobao: <div><a href="https://tengine.taobao.org">https://tengine.taobao.org</a>). Disabling buffer was a big win, but still much slower than direct HTTP access via cowboy.</div><div dir="auto"><br></div><div dir="auto">How did we find out that Nginx was the culprit?</div><div dir="auto">Simply by testing with another TLS termination proxy called Hitch (from Varnish):</div><div dir="auto"><div><a href="https://hitch-tls.org">https://hitch-tls.org</a></div><br></div><div dir="auto">Hitch is 1.5x - 2x slower than Cowboy. Unfortunately it only supports one upstream backend server at a time. Thus, we can’t serve our two webapps on port 443. Another constraint is that our two webapps has to run on the same host (a customer’s requirement).</div><div dir="auto"><br></div><div dir="auto">Finally the system is not even under load. Maximum of 10 files upload per hour. </div><div dir="auto"><br></div><div dir="auto">Forgot to mention our config:</div><div dir="auto"><br></div><div dir="auto">1. Erlang 22.1.6</div><div dir="auto">2. Linux kernel 4.15.0.66 / Ubuntu LTS 18.10 x86_64</div><div dir="auto">3. Physical machine: 32gB of RAM, 8-Cores Intel Xeon CPU E3-1270 v6@3.80GHz</div><div dir="auto">4. Nginx V1.14.0</div><div dir="auto">5. Sysctl tuned by our engineers for handle fast TCP connections with enough open files limits (ulimit -n: 200000)</div><div dir="auto"><br></div><div dir="auto">/Frank</div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le sam. 9 nov. 2019 à 03:58, Mikael Karlsson <<a href="mailto:karlsson.rm@gmail.com">karlsson.rm@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="auto">Hi,</div>Did you try with proxy_buffering set to on, and/or changing the proxy_buffer_size?<div dir="auto">Regards Mikael <br><div dir="auto"><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Den lör 9 nov. 2019 00:14Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" target="_blank">frank.muller.erl@gmail.com</a>> skrev:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><span style="color:rgb(49,49,49);word-spacing:1px">Hi guys</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">Anyone familiar with snit?</span><br style="color:rgb(49,49,49);word-spacing:1px"><a href="https://github.com/heroku/snit" rel="noreferrer noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">https://github.com/heroku/snit</a><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">We’re facing a performance issue with Nginx used as TLS Termination.</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">Nginx is in front of our two Erlang webapps. Both running on the same machine, and both based on Cowboy 2.7.0.</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">The problem:</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">[1] directly accessing the two webapps (plain HTTP) is fast enough for us, and Cowboy is doing just great.</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">[2] accessing any of the two apps with Nginx (HTTPS) is 3x-5x slower than in [1]</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">We selected Nginx for its ability to hide our apps, and be able to access them both on port 443 (default HTTPS).</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">Our Nginx config is pretty simple, tuned for SSL/TLS.</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">______________________________</span><span style="color:rgb(49,49,49);word-spacing:1px">________________</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">server {</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> listen 443 ssl;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> server_name </span><a href="http://app1.acme.com/" rel="noreferrer noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">app1.acme.com</a><span style="color:rgb(49,49,49);word-spacing:1px">; # the 2nd webapp is running on: </span><a href="http://app2.acme.com/" rel="noreferrer noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">app2.acme.com</a><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl on;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_certificate /etc/nginx/certs/app1/crt.</span><span style="color:rgb(49,49,49);word-spacing:1px">pem;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_certificate_key /etc/nginx/certs/app1/key.pem;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_dhparam /etc/nginx/certs/app1/dh.pem;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_protocols TLSv1.2;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_prefer_server_ciphers on;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_ecdh_curve</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> secp384r1</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_session_cache shared:SSL:50m;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_session_timeout 1d;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_session_tickets off;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_stapling on;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_stapling_verify on;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> resolver 8.8.8.8 8.8.4.4 valid=300s;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> resolver_timeout 5s;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> ssl_buffer_size 8k; </span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> keepalive_timeout 0;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> client_max_body_size 0;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> client_body_buffer_size 4m;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> client_header_timeout 300;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> client_body_timeout 300;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> client_header_buffer_size 1k;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> large_client_header_buffers 4 4k;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> location = /favicon.ico {</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> access_log off;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> return 204;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> }</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> location / {</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> send_timeout 5;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_http_version 1.1;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_buffering off;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_request_buffering off;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_ignore_headers "Cache-Control" "Expires";</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_max_temp_file_size 30m;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_connect_timeout 300;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_read_timeout 300;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_send_timeout 300;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_intercept_errors off;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_set_header X-Real-IP $remote_addr;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> proxy_pass </span><a href="http://127.0.0.1:2222/" rel="noreferrer noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">http://127.0.0.1:2222</a><span style="color:rgb(49,49,49);word-spacing:1px">; # the 2nd webapp has: proxy_pass </span><a href="http://127.0.0.1:3333/" rel="noreferrer noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">http://127.0.0.1:3333</a><span style="color:rgb(49,49,49);word-spacing:1px">;</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px"> }</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">}</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">______________________________</span><span style="color:rgb(49,49,49);word-spacing:1px">________________</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">Can snit be used to replace Nginx?</span><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">Help and suggestions appreciated.</span><br style="color:rgb(49,49,49);word-spacing:1px"><br style="color:rgb(49,49,49);word-spacing:1px"><span style="color:rgb(49,49,49);word-spacing:1px">Best</span></div><div><span style="color:rgb(49,49,49);word-spacing:1px">/Frank</span>
</div>
</blockquote></div>
</blockquote></div></div>