<div dir="auto">Hi Maxim,</div><div dir="auto"><br></div><div dir="auto">In my case, we are dealing with ~300 million counters per day. That’s why the new counter module didn’t fit and was also afraid to try persistent_term.</div><div dir="auto"><br></div><div dir="auto">I’m gonna look at “encount”. Thanks for sharing.</div><div dir="auto"><br></div><div dir="auto">/Frank</div><div dir="auto"><br></div><div dir="auto">Fri . 27 nov. 2020 04:42, Maxim Fedorov <<a href="mailto:dane@fb.com">dane@fb.com</a>> wrote :<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="m_-6338023644769171579WordSection1">
<p class="MsoNormal">G’Day,<br>
interestingly, I’ve been after a very similar question, and came up to this design:<u></u><u></u></p>
<p class="MsoNormal"><a href="https://github.com/max-au/ecount" target="_blank">https://github.com/max-au/ecount</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">In a gist, it’s “atomic” (could easily be swapped to “counters” but I could not prove any difference in counting performance) with counter name initially put into an ETS table, and eventually moved to “persistent_term”.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">So first attempt to find a Ref/Index is just a search in persistent_term (containing maps). If it’s not successful, then ETS table is searched, and if the name isn’t there, it’s added to ETS table (that is flushed to persistent_term every
minute).</p></div></div><div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"><div class="m_-6338023644769171579WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Maxim.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;padding:3pt 0in 0in;border-top-color:rgb(181,196,223)">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">erlang-questions <<a href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a>> on behalf of Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" target="_blank">frank.muller.erl@gmail.com</a>><br>
<b>Date: </b>Wednesday, November 25, 2020 at 6:32 AM<br>
<b>To: </b>Jesper Louis Andersen <<a href="mailto:jesper.louis.andersen@gmail.com" target="_blank">jesper.louis.andersen@gmail.com</a>><br>
<b>Cc: </b>Erlang-Questions Questions <<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
<b>Subject: </b>Re: ETS table with only counters<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks a lot Jesper!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I thought about using the new counter module, by I don’t know how many counters I will have in my system. They are created dynamically <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">/Frank<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">wed 25 nov. 2020 à 11:33, Jesper Louis Andersen <<a href="mailto:jesper.louis.andersen@gmail.com" target="_blank">jesper.louis.andersen@gmail.com</a>> wrote :<u></u><u></u></p>
<div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in;border-left-color:rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif">On Tue, Nov 24, 2020 at 8:33 PM Frank Muller <<a href="mailto:frank.muller.erl@gmail.com" target="_blank" style="font-family:Arial,sans-serif">frank.muller.erl@gmail.com</a>> wrote:<u style="font-family:Arial,sans-serif"></u><u style="font-family:Arial,sans-serif"></u></span></p>
</div>
</div>
<div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in;border-left-color:rgb(204,204,204)">
<div>
<p class="MsoNormal">Question: how should i set my table?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1. both concurrency set to true:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. only read concurrency set to true:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">3. only write concurrency set to true:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Please explain why?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif">4. Use the 'counters' module.<u style="font-family:Arial,sans-serif"></u><u style="font-family:Arial,sans-serif"></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif"><u style="font-family:Arial,sans-serif"></u> <u style="font-family:Arial,sans-serif"></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif">The caveat is that your counter set is of fixed arity and you have a way to look up an index. If both are true, they are likely to beat ETS in a measurement benchmark shootout because less locks
will be taken on them. I'd measure your results.<u style="font-family:Arial,sans-serif"></u><u style="font-family:Arial,sans-serif"></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif"><u style="font-family:Arial,sans-serif"></u> <u style="font-family:Arial,sans-serif"></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif">The answer depends on your read/write patterns for solutions 1-3. And the CPU architecture setup, mostly in core count. It's very likely you don't get a full answer by thinking about how the
locking structures are built in isolation. General rules of thumb: If mostly read, go with 2. If mostly written, go with 3. If you flip i.e., have a short window in which you mostly write and then mostly read for a while, then another write-block, ..., then
go with 1.<u style="font-family:Arial,sans-serif"></u><u style="font-family:Arial,sans-serif"></u></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif"><u style="font-family:Arial,sans-serif"></u> <u style="font-family:Arial,sans-serif"></u></span></p>
</div>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">J.<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote></div></div>