<div dir="ltr">Hi Brujo,<div><br></div><div>I tried to reproduce your results, and for me OTP 22 seems to be consistently faster for rec and about the same for lrec:</div><div><br></div><div><font face="monospace">Test case Min Max Median Average<br>rec / OTP 21 124.170us 305.990us 127.120us 128.688us<br>rec / OTP 22 62.570us 289.190us 67.610us 68.976us<br>lrec / OTP 21 16.110us 22.570us 17.790us 17.881us<br>lrec / OTP 22 15.390us 22.710us 17.170us 17.441us<br></font></div><div><br></div><div>(These are times for single invocation of the functions for much smaller inputs than in your example.)</div><div><br></div><div>However, there are some OTP 21 versions where due to a configuration bug the resolution of the native clock was much lower. For example <font face="monospace">erlang:convert_time_unit(1000, native, microsecond).</font> normally gives 1 on my machine (so the native clock has nanosecond resolution), and on the buggy OTP 21 versions I get 610. On these OTP 21 versions I see that your rec benchmark seems to run ~6% faster than with OTP 22, but this is measurement error due to the lower clock resolution.</div><div><br></div><div>Could you please check you have the same clock resolution on both OTP versions?</div><div><br></div><div>Cheers,</div><div>Daniel</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 1 Jul 2020 at 12:56, Fernando Benavides <<a href="mailto:elbrujohalcon@gmail.com">elbrujohalcon@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:"trebuchet ms",sans-serif"><div class="gmail_default">(Sorry if you receive this email twice, I sent it from another account and it seems like it didn't reach the mailing list then)</div><div class="gmail_default"><br></div><div class="gmail_default">Hi erlangers,</div><div class="gmail_default"><br></div><div class="gmail_default">Yeah! It's me talking about performance, you read that subject right 🤦♂️</div><div class="gmail_default">Anyway… At NextRoll we're in the process of migrating our systems from OTP21 to OTP22 (not 23, yet) and our tests showed a <b>huge</b> impact on performance in general, that was not associated to anything in particular.</div><div class="gmail_default"><br></div><div class="gmail_default">While trying to figure out what was causing it, we came up with a very very basic example of things that are consistently slower in OTP22.</div><div class="gmail_default"><br></div><div class="gmail_default">I created this gist to show it: <a href="https://gist.github.com/elbrujohalcon/d4e995fbc4b93fadddfd1f0d6b9f8121" style="font-family:Arial,Helvetica,sans-serif" target="_blank">https://gist.github.com/elbrujohalcon/d4e995fbc4b93fadddfd1f0d6b9f8121</a></div><div class="gmail_default"><br></div><div class="gmail_default">I'm aware that these kinds of micro-benchmarks are treacherous and they may vary wildly depending on context. Nevertheless, on both on one of our servers in AWS and on my machine (A MacOS Pro running Catalina 10.15.5) and using kerl to install multiple versions of OTP 21 and 22… every single time I run the tests I found the same results…</div><div class="gmail_default"><br></div><div class="gmail_default">Always starting the nodes with <i>erl -boot start_clean </i>then running the following in both OTP21 and OTP22…</div><div class="gmail_default" style="font-family:Arial,Helvetica,sans-serif"><font face="monospace"><br></font></div><div class="gmail_default" style="font-family:Arial,Helvetica,sans-serif"><font face="monospace">c(test), test:bench({test, lrec}, 250, 5000, 2000).</font><br></div><div class="gmail_default"><br></div><div class="gmail_default">…generates very similar numbers in both versions, regardless of the numbers used for the different parameters. But…</div><div class="gmail_default" style="font-family:Arial,Helvetica,sans-serif"><font face="monospace"><br></font></div><div class="gmail_default" style="font-family:Arial,Helvetica,sans-serif"><font face="monospace">c(test), test:bench({test, rec}, 250, 5000, 2000).</font><br></div><div class="gmail_default"><br></div><div class="gmail_default">…consistently generates larger results in OTP22 than OTP21. I tried with different values for the number of tests, the number of lists and their length and sometimes the difference is more evident, sometimes less… but OTP22 times are <b>always</b> larger.</div><div class="gmail_default"><br></div><div class="gmail_default">I found this in the OTP22 readme…</div><div class="gmail_default"><br></div><div class="gmail_default" style="font-family:Arial,Helvetica,sans-serif"><font face="monospace"> OTP-15427 Application(s): erts<br> Appending lists (The ++ operator) will now yield<br> properly on large inputs.</font><br></div><div class="gmail_default"><br></div><div class="gmail_default">So… questions…</div><div class="gmail_default">1. Has anybody experienced (and hopefully solved) this problem before when migrating to OTP22?</div><div class="gmail_default">2. Do you think OTP-15427 can be related to what I'm seeing?</div><div class="gmail_default">3. Can someone confirm if you also experience the same difference in performance when running the same benchmarks that I pasted on that gist?</div><div class="gmail_default">4. Is there anything else I should try/test/use to check?</div><div class="gmail_default">5. Am I going slightly mad?</div><div class="gmail_default"><br></div><div class="gmail_default">Thanks in advance, cheers :)</div><font color="#888888" style="font-family:Arial,Helvetica,sans-serif"><br></font></div><div><br></div>-- <br><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td align="left" valign="bottom" width="107" style="line-height:0;vertical-align:bottom;padding-right:10px;padding-top:20px;padding-bottom:20px">
<a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none" target="_blank">
<img src="https://thumbs.about.me/thumbnail/users/e/l/b/elbrujohalcon_emailsig.jpg?_1498084031_146" alt="" width="105" height="70" style="margin: 0px; padding: 0px; display: block; border: 1px solid rgb(238, 238, 238);">
</a>
</td>
<td align="left" valign="bottom" style="line-height:1.1;vertical-align:bottom;padding-top:20px;padding-bottom:20px">
<img src="https://about.me/t/sig?u=elbrujohalcon" width="1" height="1" style="border: 0px; margin: 0px; padding: 0px; overflow: hidden;">
<div style="font-size:18px;font-weight:bold;color:rgb(51,51,51);font-family:"Proxima Nova",Helvetica,Arial,sans-serif">Brujo Benavides</div>
<a href="https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb" style="text-decoration:none;font-size:12px;color:rgb(43,130,173);font-family:"Proxima Nova",Helvetica,Arial,sans-serif" target="_blank">about.me/elbrujohalcon
</a>
</td>
</tr>
</tbody>
</table>
</div></div>
</blockquote></div>