<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 26 Aug 2020, at 17:01, Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" class="">frank.muller.erl@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div dir="auto" class="">Hi Pablo,</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Would be great if you can share here how did you narrow down the issue to Madvice call. </div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Everyone in the mailinglist will benefit from it.</div><div dir="auto" class=""><br class=""></div></div></div></blockquote><div>Hi, sure.</div><div>We hit a bad drop on throughput when tried to migrate from OTP21 to OTP22. Initially we checked the changelogs looking for anything suspicious</div><div>(like changes on default behaviours/settings), but that didn’t point to anything obvious.</div><div>We noticed that the system CPU usage was much higher. Looked into that more closely and noted that the # of page faults skyrockets on the new</div><div>version vs the old one:</div><div><div><br class=""></div><div>$ sudo sar -B 10 10</div><div>Linux 4.14.177-107.254.amzn1.x86_64 (ip-172-26-68-64) <span class="Apple-tab-span" style="white-space:pre"> </span>08/03/2020 <span class="Apple-tab-span" style="white-space:pre"> </span>_x86_64_<span class="Apple-tab-span" style="white-space:pre"> </span>(16 CPU)</div><div>09:21:55 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff</div><div>09:22:05 PM 0.00 243.64 140891.62 0.00 128650.61 0.00 0.00 0.00 0.00</div><div>09:22:15 PM 0.00 103.24 135143.62 0.00 129283.20 0.00 0.00 0.00 0.00</div><div class="">[…]</div><div class=""><br class=""></div><div class="">Note the huge # of faults (and those are minor page faults, so not like the node was trashing out of memory).</div><div class=""><br class=""></div><div class="">With that, monitored for a couple seconds the memory related syscalls that we where doing vs what</div><div class="">was doing the previous version, to check if there was a different pattern. What we found was the madvice one,</div><div class="">that didn’t appear before:</div><div class=""><div class=""><br class=""></div><div class="">$ sudo strace -f -e trace=memory -c -p $pid</div><div class=""><br class=""></div><div class="">% time seconds usecs/call calls errors sys-call</div><div class="">------ ----------- ----------- --------- --------- ----------------</div><div class=""> 86.91 0.622994 74 8371 madvise <- this is the smoking gun</div><div class=""> 7.39 0.053005 71 746 munmap</div><div class=""> 5.70 0.040851 62 656 mmap</div><div class="">------ ----------- ----------- --------- --------- ----------------</div></div><div class=""><br class=""></div><div class="">That lead us to </div><div class=""><a href="https://github.com/erlang/otp/pull/2046" class="">https://github.com/erlang/otp/pull/2046</a> (listed here <a href="http://blog.erlang.org/OTP-22-Highlights/" class="">http://blog.erlang.org/OTP-22-Highlights/</a>) </div><div class=""><br class=""></div><div class="">Take a look at</div><div class=""><a href="https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55" class="">https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55</a></div><div class="">a classic, short, concise, and incredibly useful guide on what to check when you start digging for performance</div><div class="">problems, before trying to look at your code. </div><div class=""><br class=""></div><div class="">cheers</div></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><div dir="auto" class="">Thanks</div><div dir="auto" class="">/Frank</div><div dir="auto" class=""><br class=""></div><div dir="auto" class=""><br class=""></div></div><div class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr" class=""><div dir="auto" class=""><div class=""><br class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 25 Aug 2020, 17:28 Pablo Polvorin, <<a href="mailto:pablo.polvorin@nextroll.com" target="_blank" class="">pablo.polvorin@nextroll.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div style="word-wrap:break-word;line-break:after-white-space" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 25 Aug 2020, at 09:05, Lukas Larsson <<a href="mailto:lukas@erlang.org" rel="noreferrer" target="_blank" class="">lukas@erlang.org</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><div style="font-family:Helvetica" class="">Hello,</div><br class=""><div class="gmail_quote" style="font-family:Helvetica"><div dir="ltr" class="gmail_attr" style="font-family:Helvetica">On Tue, Aug 25, 2020 at 9:26 AM Pablo Polvorin <<a href="mailto:pablo.polvorin@nextroll.com" rel="noreferrer" target="_blank" style="font-family:Helvetica" class="">pablo.polvorin@nextroll.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">Hi,<br class="">we recently migrated our systems from OTP21 to OTP22. On one of them, performance noticeable degraded when switched to OTP22 (20% drop).<br class="">After some investigation, we found the apparent cause was the madvice() call when returning the carriers to the carrier pool, that lead to lots of<br class="">minor page faults, ultimately killing our performance. We run in a virtualised cloud, not sure how much that affects the minor page fault overhead.<br class=""></blockquote><div style="font-family:Helvetica" class=""><br class=""></div><div style="font-family:Helvetica" class="">Do you know if you have access to MADV_FREE or use MADV_DONTNEED?</div><div style="font-family:Helvetica" class=""><br class=""></div></div></div></div></blockquote><div class="">Looks like we don’t :/ </div><div class=""><div class=""><br class=""></div><div class="">#include <sys/mman.h></div><div class="">#include <stdio.h></div><div class="">int main() {</div><div class=""> #ifdef MADV_FREE</div><div class=""><span style="white-space:pre-wrap" class=""> </span> printf("Have it\n");</div><div class=""> #else</div><div class=""><span style="white-space:pre-wrap" class=""> </span> printf("Dont have it\n");</div><div class=""> #endif</div><div class=""> return 0;</div><div class="">}</div></div><div class="">> <span style="font-family:Menlo" class="">Dont have it</span></div><div class=""><br class=""></div><div class="">This is on amazon linux, 4.14.186-110.268.amzn1.x86_64 .</div><div class=""></div></div></div></blockquote></div></div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">That may explain it. Maybe we should not use madvise when FREE is not available.</div><div dir="auto" class=""><br class=""></div><div class="">Have you tried do delete this line: <a href="https://github.com/erlang/otp/blob/7ad81c674d1aa705ae41743b343043d05ea1944b/erts/emulator/sys/common/erl_mmap.h#L215" target="_blank" class="">https://github.com/erlang/otp/blob/7ad81c674d1aa705ae41743b343043d05ea1944b/erts/emulator/sys/common/erl_mmap.h#L215</a> and see what happens then?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div style="word-wrap:break-word;line-break:after-white-space" class=""><div class=""><div class=""><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><div class="gmail_quote" style="font-family:Helvetica"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)"><br class="">So we spend some time tweaking the allocator' settings (that haven't been touched in many years, with the system itself evolved a lot since that time) and got<br class="">to a good improvement. But for carrier migration ultimately the thing that worked best for us was just disable it entirely. Is this a terrible idea?<br class="">our load is a fairly stable flow of homogenous requests, that lead to several short-lived (milliseconds) processes being spawned. Have plenty of memory, so I'm not too worried about a badly utilised carrier being stuck within a scheduler.<br class=""><br class="">Got a few questions regarding this:<br class=""><br class="">* Wonder, it's something common out there to disable carrier migration? Feels a bit strange that nobody hit the same problem when updating to OTP22, I’m<br class="">assuming there are lots of not-so-great allocator settings out there, like was our case. (disabling it was our last try actually, we tried the settings suggested by<br class="">erts_alloc_config, and then make the +M<S>acul and +M<S>acfml settings significantly stricter as well, and while that helps, still had too many page faults).<br class=""><br class=""></blockquote><div style="font-family:Helvetica" class=""><br class=""></div><div style="font-family:Helvetica" class="">Carrier migration can help a lot to deal with memory fragmentation issues. It is however not free as you have noticed. I know that other people have disabled it with some success, but as far as I remember that was to work around bugs in the migration logic, not because of the performance overhead.</div></div></div></div></blockquote><div class="">Given the frequency at which we where abandoning and taking carriers from the pool, smells something fishy on our settings and allocation pattern. But so far couldn’t really figure out how to bring that down to a low enough level that the madvice() won’t affect us much.</div></div></div></blockquote></div></div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Yes, that does seem odd. Carriers are not meant to be pushed in and put of the pool at a rapid pace.</div><div dir="auto" class=""><br class=""></div><div class="">I don't suppose you have a relatively small example that will reproduce the behaviour? Or if not, then maybe a couple of recon_alloc snapshots?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div style="word-wrap:break-word;line-break:after-white-space" class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><div class="gmail_quote" style="font-family:Helvetica"><div style="font-family:Helvetica" class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">* What's the tradeoff on having large multiblock carriers, other than the memory overhead when they aren't fully used?.<br class=""><br class="">* Do it make sense to make a config flag to allow carrier migration but disable the madvice() on free blocks?<br class=""></blockquote><div style="font-family:Helvetica" class=""><br class=""></div><div style="font-family:Helvetica" class="">Given your experiences with it, I think that would make sense. We did not notice any degradation when testing madvise ourselves, but it is not possible to test all scenarios in all environments.</div><div style="font-family:Helvetica" class=""><br class=""></div></div></div></div></blockquote><div class="">Might work on this, although I guess it would require to re-learn autoconf sorcery so probably will not happen soon </div></div></div></blockquote><div class=""><br class=""></div><div class="">No need to do any autoconf sorcery, I was thinking that this could probably be a start flag passed to erl?</div></div></div></div><br class=""><br class=""></div><br class=""><br class=""></blockquote></div></div>
</div></blockquote></div><br class=""></body></html>