<div dir="ltr"><div>Hello,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 3, 2020 at 4:14 PM Fernando Benavides <<a href="mailto:elbrujohalcon@gmail.com">elbrujohalcon@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"trebuchet ms",sans-serif">Hi Erlangers,</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">It's me again… still trying to understand why my system behaves worse in OTP22 than OTP21.</div><div style="font-family:"trebuchet ms",sans-serif">The last thing I tried was running <a href="http://erlang.org/doc/man/msacc.html" target="_blank">MicroStateAccounting</a> on it and the results showed something <i>odd</i>…</div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div style="font-family:"trebuchet ms",sans-serif">These are two of my summaries for OTP21:</div><div><font size="1" face="monospace">Stats per type:<br>        Thread      <b>aux</b> check_io <b>emulator</b>       gc    other     port    sleep<br>         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%<br>           aux    0.23%    0.06%    0.00%    0.00%    0.09%    0.00%   99.62%<br>dirty_cpu_sche    0.00%    0.00%    0.01%    0.20%    0.00%    0.00%   99.79%<br>dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%<br>          poll    0.00%    0.58%    0.00%    0.00%    0.00%    0.00%   99.42%<br>     scheduler    <b>2.05%</b>    0.56%   <b>71.29%</b>   13.25%    3.00%    1.89%    7.96%<br>ok<br><br>Stats per type:<br>         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%<br>           aux    0.23%    0.07%    0.00%    0.00%    0.08%    0.00%   99.62%<br>dirty_cpu_sche    0.00%    0.00%    0.01%    0.29%    0.00%    0.00%   99.70%<br>dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%<br>          poll    0.00%    0.61%    0.00%    0.00%    0.00%    0.00%   99.39%<br>     <b>scheduler</b>    <b>2.26%</b>    0.56%   <b>68.70%</b>   12.69%    3.42%    1.89%   10.48%<br>ok</font><br></div><div style="font-family:"trebuchet ms",sans-serif"><br></div><div><div style="font-family:"trebuchet ms",sans-serif">These are the corresponding two summaries for OTP22 (same period of time, same code, running the same things):</div><div><font size="1" face="monospace">Stats per type:<br>        Thread      <b>aux</b> check_io <b>emulator</b>       gc    other     port    sleep<br>         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%<br>           aux    0.49%    0.06%    0.00%    0.00%    0.28%    0.00%   99.16%<br>dirty_cpu_sche    0.00%    0.00%    0.00%    0.19%    0.00%    0.00%   99.80%<br>dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%   99.99%<br>          poll    0.00%    0.53%    0.00%    0.00%    0.00%    0.00%   99.47%<br>     <b>scheduler</b>    <b>6.26%</b>    0.52%   <b>63.83%</b>   14.41%    3.05%    1.72%   10.20%<br>ok</font></div><div><font size="1" face="monospace"><br>Stats per type:<br>         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%<br>           aux    0.39%    0.07%    0.00%    0.00%    0.18%    0.00%   99.36%<br>dirty_cpu_sche    0.00%    0.00%    0.00%    0.22%    0.00%    0.00%   99.78%<br>dirty_io_sched    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%   99.99%<br>          poll    0.00%    0.51%    0.00%    0.00%    0.00%    0.00%   99.49%<br>     <b>scheduler</b>    <b>6.97%</b>    0.54%   <b>63.88%</b>   13.73%    3.19%    1.74%    9.96%<br>ok</font><br></div></div><div><br></div><div><div style="font-family:"trebuchet ms",sans-serif">I highlighted the strange/different parts there, but in a nutshell:</div><div style="font-family:"trebuchet ms",sans-serif">The scheduler is spending more time <i>handling </i><b><i>auxiliary jobs</i> </b>and less time <i>executing Erlang processes </i>(<a href="http://erlang.org/doc/man/erlang.html#statistics_microstate_accounting" target="_blank">for context</a>).</div><div style="font-family:"trebuchet ms",sans-serif">Can anybody give me a clue as to what to check next? How can I determine what are those <i><b>auxiliary jobs</b></i> that the scheduler is working on now?</div><div style="font-family:"trebuchet ms",sans-serif"><br></div></div></div></blockquote><div><br></div><div>You can enable more msacc stats by turning on extra states. This is a compile-time flag that you pass to configure: ./configure --with-microstate-accounting=extra</div><div><br></div><div>My guess would be that the extra time ends up in the alloc state, which means that it for some reason extra memory allocation is happening which is slowing down the system.</div><div><br></div><div>Lukas</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="font-family:"trebuchet ms",sans-serif"></div><div style="font-family:"trebuchet ms",sans-serif">Thanks in advance :)</div><br></div><div><br></div>-- <br><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0">
    <tbody>
        <tr>
            <td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
                    <img alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
                </a>
            </td>
            <td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
                <img width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
                <div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
                <a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
                </a>
            </td>
        </tr>
    </tbody>
</table>
</div></div>
</blockquote></div></div>