<div dir="ltr"><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I'll add another data point…</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">As Lukas suggested, I used <i>perf</i> on the running systems and this is what I've got…</div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">For OTP22:</div><div class="gmail_default" style=""><font face="monospace">     2.33%  8_scheduler      beam.smp             [.] process_main<br><i>     …the rest of the schedulers running process_main…<br></i>     1.57%  6_scheduler      beam.smp             [.] process_main<br>     0.34%  6_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.32%  7_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.25%  14_scheduler     beam.smp             [.] copy_struct_x<br>     0.25%  2_scheduler      beam.smp             [.] copy_struct_x<br>     0.25%  9_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.23%  11_scheduler     [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.22%  11_scheduler     beam.smp             [.] copy_shallow_x<br>     0.22%  9_scheduler      <a href="http://libc-2.17.so">libc-2.17.so</a>         [.] __memcpy_ssse3_back<br>     0.20%  13_scheduler     [kernel.kallsyms]    [k] __lock_text_start<br>     0.20%  3_scheduler      [kernel.kallsyms]    [k] xen_hypercall_event_channel_op<br>     0.19%  10_scheduler     beam.smp             [.] erts_cmp_compound<br></font></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">For OTP21:</div><div class="gmail_default" style=""><font face="monospace">     2.45%  4_scheduler      beam.smp             [.] process_main<br><i style="">     …the rest of the schedulers running process_main…<br></i>     1.91%  5_scheduler      beam.smp             [.] process_main<br>     0.45%  15_scheduler     [kernel.kallsyms]    [k] __lock_text_start<br>     0.27%  10_scheduler     [kernel.kallsyms]    [k] __lock_text_start<br>     0.27%  3_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.27%  7_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.25%  1_scheduler      beam.smp             [.] copy_struct_x<br>     0.25%  2_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.23%  2_scheduler      beam.smp             [.] copy_struct_x<br>     0.23%  7_scheduler      beam.smp             [.] copy_struct_x<br>     0.20%  4_scheduler      beam.smp             [.] copy_struct_x<br>     0.20%  6_scheduler      [kernel.kallsyms]    [k] __lock_text_start<br>     0.18%  14_scheduler     beam.smp             [.] copy_struct_x</font><br></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="trebuchet ms, sans-serif">I'm not sure what any of those functions are, or why </font><font face="monospace">xen_hypercall_event_channel_op</font><font face="trebuchet ms, sans-serif"> is so much popular in OTP22, tho… 🤔</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 3:42 PM Max Lapshin <<a href="mailto:max.lapshin@gmail.com">max.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In a nutshell there is a non-reproducible fluctuation of measurement:  one subsystem takes 10% instead 8%, another 21% instead 23%.<div><br></div><div>Totally there seems to be no change.</div><div><br></div><div>It is extremely hard to make reliable measurement that helps to find 8% to 10% change.</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><table border="0" cellpadding="0" cellspacing="0">
    <tbody>
        <tr>
            <td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
                    <img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
                </a>
            </td>
            <td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
                <img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
                <div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
                </a>
            </td>
        </tr>
    </tbody>
</table>
</div>