<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">You should run on the same machines, and preferably at the same time (back to back). When you are running under a hypervisor, you are cooperating with other users of the hardware, and they can wreak serious havoc on your benchmarks. I've seen 20% fluctuations in performance on AWS hardware due to this. The usual problems stems from how the CPUs handle caches and if there are other users who move a lot of data around, you will see a lower performance due to this.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">My advice will be to reconstruct the problem on local hardware.<i> </i>If you cannot do this, then chances are you are looking at some AWS artifact rather than a change in OTP behavior.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 5:14 PM Fernando Benavides <<a href="mailto:elbrujohalcon@gmail.com">elbrujohalcon@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default"><font face="trebuchet ms, sans-serif">Yeah… the curious thing is that both machines are the same kind of AWS virtual machines… And I had 2 running OTP22 and 2 running OTP21… and both OTP22 machines showed </font><span style="font-family:monospace">xen_hypercall_event_</span><span style="font-family:monospace">channel_op </span><span style="font-family:Arial,Helvetica,sans-serif">while both OTP21 didn't (actually… they did, but with a very smaller % of overhead). And, BTW, the same happened with </span><font face="monospace">smp_call_function_many</font><span style="font-family:Arial,Helvetica,sans-serif"> (more overhead associated with that function in OTP22 than OTP21).</span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif"><br></span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif">I'll keep digging, but if anybody has any idea of what to look for, I would appreciate it a lot.</span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif"><br></span></div><div class="gmail_default"><span style="font-family:Arial,Helvetica,sans-serif">Thanks</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 5:05 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com" target="_blank">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">First post from the internet about <span style="font-family:monospace">xen_hypercall_event_channel_op:</span> <a href="https://www.scylladb.com/2018/06/21/impact-virtualization-database/" target="_blank">https://www.scylladb.com/2018/06/21/impact-virtualization-database/</a><div><br></div><div>TL;DR: metal server better than virtual</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 5:26 PM Fernando Benavides <<a href="mailto:elbrujohalcon@gmail.com" target="_blank">elbrujohalcon@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">I'll add another data point…</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">As Lukas suggested, I used <i>perf</i> on the running systems and this is what I've got…</div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">For OTP22:</div><div class="gmail_default"><font face="monospace"> 2.33% 8_scheduler beam.smp [.] process_main<br><i> …the rest of the schedulers running process_main…<br></i> 1.57% 6_scheduler beam.smp [.] process_main<br> 0.34% 6_scheduler [kernel.kallsyms] [k] xen_hypercall_event_channel_op<br> 0.32% 7_scheduler [kernel.kallsyms] [k] xen_hypercall_event_channel_op<br> 0.25% 14_scheduler beam.smp [.] copy_struct_x<br> 0.25% 2_scheduler beam.smp [.] copy_struct_x<br> 0.25% 9_scheduler [kernel.kallsyms] [k] xen_hypercall_event_channel_op<br> 0.23% 11_scheduler [kernel.kallsyms] [k] xen_hypercall_event_channel_op<br> 0.22% 11_scheduler beam.smp [.] copy_shallow_x<br> 0.22% 9_scheduler <a href="http://libc-2.17.so" target="_blank">libc-2.17.so</a> [.] __memcpy_ssse3_back<br> 0.20% 13_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.20% 3_scheduler [kernel.kallsyms] [k] xen_hypercall_event_channel_op<br> 0.19% 10_scheduler beam.smp [.] erts_cmp_compound<br></font></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif"><br></div><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif">For OTP21:</div><div class="gmail_default"><font face="monospace"> 2.45% 4_scheduler beam.smp [.] process_main<br><i> …the rest of the schedulers running process_main…<br></i> 1.91% 5_scheduler beam.smp [.] process_main<br> 0.45% 15_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.27% 10_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.27% 3_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.27% 7_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.25% 1_scheduler beam.smp [.] copy_struct_x<br> 0.25% 2_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.23% 2_scheduler beam.smp [.] copy_struct_x<br> 0.23% 7_scheduler beam.smp [.] copy_struct_x<br> 0.20% 4_scheduler beam.smp [.] copy_struct_x<br> 0.20% 6_scheduler [kernel.kallsyms] [k] __lock_text_start<br> 0.18% 14_scheduler beam.smp [.] copy_struct_x</font><br></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">I'm not sure what any of those functions are, or why </font><font face="monospace">xen_hypercall_event_channel_op</font><font face="trebuchet ms, sans-serif"> is so much popular in OTP22, tho… 🤔</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 3:42 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com" target="_blank">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In a nutshell there is a non-reproducible fluctuation of measurement: one subsystem takes 10% instead 8%, another 21% instead 23%.<div><br></div><div>Totally there seems to be no change.</div><div><br></div><div>It is extremely hard to make reliable measurement that helps to find 8% to 10% change.</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
<a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
<img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
</a>
</td>
<td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
<img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
<div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
<a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
</a>
</td>
</tr>
</tbody>
</table>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
<a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
<img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
</a>
</td>
<td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
<img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
<div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
<a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
</a>
</td>
</tr>
</tbody>
</table>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">J.</div>