<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Just my 2c (for what is) worth…<div class=""><br class=""></div><div class="">There are some versions of Linux kernel that are known to cause performance problems. Recently, we had a situation where a particular OS kernel patch caused a serious performance degradation. We were lucky, as the same software ran on multiple machines with almost the same hardware configuration, thus we did not have to doubt Erlang, or our code to that matter. </div><div class="">BTW: the worst performing machine was running at 30% capacity relative to the best performing machine. After a brief wild goose chase with a hardware vendor, they (the vendor) have established that the problem was not related to the Intel CPU (Speedstep Technology), as *we* initially suspected, but actual OS kernel version.</div><div class=""><br class=""></div><div class="">When you become desperate enough, you could check your OS kernel version/patch level as well.</div><div class=""><br class=""></div><div class="">V/ <br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 03 Jan 2020, at 19:39, Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" class="">frank.muller.erl@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><span style="color:rgb(49,49,49);word-spacing:1px" class="">Hi Daniel</span><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Thanks for the feedback. Sorry for being unclear, but this thread discontinued when switched to 2020. I gave some context when I started it at:</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><a href="http://erlang.org/pipermail/erlang-questions/2019-December/098910.html" target="_blank" style="font-size:1rem" class="">http://erlang.org/pipermail/erlang-questions/2019-December/098910.html</a></div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">More context:</div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">1. I’m building a Hugo (<a href="https://gohugo.io/" target="_blank" style="font-size:1rem" class="">https://gohugo.io/</a>) like service in Erlang.</div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Actually, it’s much more simpler than Hugo. You ask for a file, and you get it (or not).</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">2. The design is simple. One Erlang process handles one incoming connection.</div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Thus, it’s not possible to mess up with TCP sockets here. Whether the file exists and gets delivered to the caller or not.</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">3. Average file size is ~150KB. Max file size: ~3MB</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">4. I’m using Erlang 22.2 on both Linux (prod system) and MacOS (dev).</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">5. The service is caping at 4'000 req/sec.</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">6. @Mikael Pettersson suggested to use prim_file:sendfile/8 which gaves us a boost of ~3% in throughput.</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">7. For now, I’m only using HTTP (no HTTPS).</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">8. Linux sysctl settings are good and tuned (<a href="https://medium.com/@pawilon/tuning-your-linux-kernel-and-haproxy-instance-for-high-loads-1a2105ea553e" target="_blank" style="font-size:1rem" class="">https://medium.com/@pawilon/tuning-your-linux-kernel-and-haproxy-instance-for-high-loads-1a2105ea553e</a>):</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><div style="font-size:1rem" class="">$ ulimit -n</div><div style="font-size:1rem" class="">655360</div></div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Anything else you’d like to know?</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">I completely agree with you regarding the Port locks. But when 1 collision occurs, it tooks 72’690'401usec.</div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">That’s my concern :-/</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Can these Port locks be avoided?</div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Is there any undocumented feature I can try?</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">I’m open to test any idea and/or change my design architecture if needed. </div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">Thanks in advance.</div><div style="color:rgb(49,49,49);word-spacing:1px" dir="auto" class=""><br class=""></div><div style="font-size:1rem;color:rgb(49,49,49);word-spacing:1px" dir="auto" class="">/Frank</div><br class="Apple-interchange-newline" style="-webkit-tap-highlight-color: rgba(26, 26, 26, 0.301961); -webkit-text-size-adjust: auto;"></div><div class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Fri 3 jan 2020 at 10:21, Dániel Szoboszlay <<a href="mailto:dszoboszlay@gmail.com" class="">dszoboszlay@gmail.com</a>> wrote :<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Hi Frank,<div class=""><br class=""></div><div class="">I think some context would help us to better understand your problem. For example what kind of problems do these lock conflicts cause in your system? Are there some Erlang processes that got stuck for several seconds? What are they trying to do at that time? Or is it only that the throughput you experience stays below your expectations?</div><div class=""><br class=""></div><div class="">The lock stats in the gists don't seem that wrong to me. I don't have much experience with lock counters and can't figure out from the gists what do these ports are (the image files? tcp connections?) and how they are used, but my guess is they are the tcp connections. Each port lock is tried 4 or 5 times and there's only 1 (albeit often very long) collision. This is how I'd expect a web server to handle a tcp connection: do one read, one write, close it - a handful of operations only. And if you have an unresponsive client or bad network, one operation may easily take 6-7 seconds, blocking the next one. So just a guess, but is it possible you attempt to use the same tcp socket from two different processes?</div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Daniel</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 3 Jan 2020 at 00:40, Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" target="_blank" class="">frank.muller.erl@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div dir="auto" class="">The more load we have, the more locks contention we get:</div></div><div dir="auto" class=""><br class=""></div><div class=""><div class=""><a href="https://gist.github.com/frankmullerl/7fb9470e22869312d97011c0faf0046b" target="_blank" class="">https://gist.github.com/frankmullerl/7fb9470e22869312d97011c0faf0046b</a></div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">/Frank</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr"><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto" class="">Hi Mikael</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Thanks for pointing out prim_inet. To my knowledge, it’s undocumented I can be changed by the OTP team. That’s why I’ve avoided it at the first place. </div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">By the way, you probably mean prim_inet:sendfile/4 not 8, correct?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">I made a quick the change to my app and found that the throughput is a little bit better (3% faster).</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">But these port locks are still there :-/</div><div dir="auto" class="">Any other ideas?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">/Frank</div><div dir="auto" class=""><br class=""></div><div class="">On Thu, Jan 2, 2020 at 5:38 PM Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" target="_blank" class="">frank.muller.erl@gmail.com</a>> wrote:<br class=""></div><div class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
><br class="">
> Thanks to @Peti Gömöri, I was able to identify these Port locks:<br class="">
> <a href="https://gist.github.com/frankmullerl/008174c6594ca27584ac7f4e6724bee5" rel="noreferrer" target="_blank" class="">https://gist.github.com/frankmullerl/008174c6594ca27584ac7f4e6724bee5</a><br class="">
><br class="">
> Some of them are taking up to 6.7sec (6707846 usec) :-/<br class="">
><br class="">
> My application is serving static images by calling file:sendfile/2 (<a href="https://erlang.org/doc/man/file.html#sendfile-2" rel="noreferrer" target="_blank" class="">https://erlang.org/doc/man/file.html#sendfile-2</a>).<br class="">
><br class="">
> Can someone please explain how I can avoid these locks or at least make their impact lesser?<br class="">
<br class="">
Using the file module has been known to cause synchronization<br class="">
overheads. Often we (Klarna) use prim_file instead,<br class="">
but it has sendfile/8 not /2. Anyway, that may be an avenue worth<br class="">
investigating.<br class="">
</blockquote></div></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></div></body></html>