<div dir="ltr"><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Thanks Jesper,</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I understand all that but I already run the tests in more than 32 different AWS machines… in groups of 2 OTP22 / 2 OTP21 machines each time… and the results are impressively consistent. Every single time I get the results I posted before (different %s of course, but not significantly different and it's always the same difference ratio between the different OTP versions).</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I'm really really convinced that this is not a fluke due to virtualization. It's something that our system is doing that got worse in OTP22, for sure… but I don't know where to look.</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Cheers!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 7, 2020 at 12:40 PM Jesper Louis Andersen <<a href="mailto:jesper.louis.andersen@gmail.com">jesper.louis.andersen@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">You should run on the same machines, and preferably at the same time (back to back). When you are running under a hypervisor, you are cooperating with other users of the hardware, and they can wreak serious havoc on your benchmarks. I've seen 20% fluctuations in performance on AWS hardware due to this. The usual problems stems from how the CPUs handle caches and if there are other users who move a lot of data around, you will see a lower performance due to this.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">My advice will be to reconstruct the problem on local hardware.<i> </i>If you cannot do this, then chances are you are looking at some AWS artifact rather than a change in OTP behavior.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 5:14 PM Fernando Benavides <<a href="mailto:elbrujohalcon@gmail.com" target="_blank">elbrujohalcon@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default"><font face="trebuchet ms, sans-serif">Yeah… the curious thing is that both machines are the same kind of AWS virtual machines… And I had 2 running OTP22 and 2 running OTP21… and both OTP22 machines showed </font><span style="font-family:monospace">xen_hypercall_event_</span><span style="font-family:monospace">channel_op </span><span style="font-family:Arial,Helvetica,sans-serif">while both OTP21 didn't (actually… they did, but with a very smaller % of overhead). And, BTW, the same happened with </span><font face="monospace">smp_call_function_many</font><span style="font-family:Arial,Helvetica,sans-serif"> (more overhead associated with that function in OTP22 than OTP21).</span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif"><br></span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif">I'll keep digging, but if anybody has any idea of what to look for, I would appreciate it a lot.</span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif"><br></span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif">Thanks</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 5:05 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com" target="_blank">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">First post from the internet about <span style="font-family:monospace">xen_hypercall_event_channel_op:</span>  <a href="https://www.scylladb.com/2018/06/21/impact-virtualization-database/" target="_blank">https://www.scylladb.com/2018/06/21/impact-virtualization-database/</a><div><br></div><div>TL;DR: metal server better than virtual</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 5:26 PM Fernando Benavides <<a href="mailto:elbrujohalcon@gmail.com" target="_blank">elbrujohalcon@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">I'll add another data point…</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">As Lukas suggested, I used <i>perf</i> on the running systems and this is what I've got…</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">For OTP22:</div><div class="gmail_default"><font face="monospace">     2.33%  8_scheduler      beam.smp             [.] process_main<br><i>     …the rest of the schedulers running process_main…<br></i>     1.57%  6_scheduler      beam.smp             [.] process_main<br>     0.34%  6_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.32%  7_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.25%  14_scheduler     beam.smp             [.] copy_struct_x<br>     0.25%  2_scheduler      beam.smp             [.] copy_struct_x<br>     0.25%  9_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.23%  11_scheduler     [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.22%  11_scheduler     beam.smp             [.] copy_shallow_x<br>     0.22%  9_scheduler      <a href="http://libc-2.17.so" target="_blank">libc-2.17.so</a>         [.] __memcpy_ssse3_back<br>     0.20%  13_scheduler     [kernel.kallsyms]    [k] __lock_text_start<br>     0.20%  3_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.19%  10_scheduler     beam.smp             [.] erts_cmp_compound<br></font></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">For OTP21:</div><div class="gmail_default"><font face="monospace">     2.45%  4_scheduler      beam.smp             [.] process_main<br><i>     …the rest of the schedulers running process_main…<br></i>     1.91%  5_scheduler      beam.smp             [.] process_main<br>     0.45%  15_scheduler     [kernel.kallsyms]    [k] __lock_text_start<br>     0.27%  10_scheduler     [kernel.kallsyms]    [k] __lock_text_start<br>     0.27%  3_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.27%  7_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.25%  1_scheduler      beam.smp             [.] copy_struct_x<br>     0.25%  2_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.23%  2_scheduler      beam.smp             [.] copy_struct_x<br>     0.23%  7_scheduler      beam.smp             [.] copy_struct_x<br>     0.20%  4_scheduler      beam.smp             [.] copy_struct_x<br>     0.20%  6_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.18%  14_scheduler     beam.smp             [.] copy_struct_x</font><br></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">I'm not sure what any of those functions are, or why </font><font face="monospace">xen_hypercall_event_channel_op</font><font face="trebuchet ms, sans-serif"> is so much popular in OTP22, tho… 🤔</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 3:42 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com" target="_blank">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In a nutshell there is a non-reproducible fluctuation of measurement:  one subsystem takes 10% instead 8%, another 21% instead 23%.<div><br></div><div>Totally there seems to be no change.</div><div><br></div><div>It is extremely hard to make reliable measurement that helps to find 8% to 10% change.</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0">
    <tbody>
        <tr>
            <td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
                    <img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
                </a>
            </td>
            <td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
                <img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
                <div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
                </a>
            </td>
        </tr>
    </tbody>
</table>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0">
    <tbody>
        <tr>
            <td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
                    <img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
                </a>
            </td>
            <td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
                <img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
                <div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
                </a>
            </td>
        </tr>
    </tbody>
</table>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">J.</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><table border="0" cellpadding="0" cellspacing="0">
    <tbody>
        <tr>
            <td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
                    <img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
                </a>
            </td>
            <td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
                <img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
                <div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
                </a>
            </td>
        </tr>
    </tbody>
</table>
</div>