improving match_object for ets tables?

[Sverker Eriksson]

On fre, 2020-02-21 at 15:44 +0000, Vans S wrote:
Anyone have any ideas for improvements in mind to mach_object, perhaps maybe a way to have it provide 0 consistency?

It seems to be a reoccurring problem, the code runs great locally with light load. As soon as you scale up, 64+ cores 50k+ sessions. All of the sudden your schedulers are capped out at 100% utilization, runqueues are maxing out, but the physical cpu usage is 50%<.

This is because match_object waits for locks on the table and does not yield the scheduler.  As soon as the concurrent access increases to the table, the problem exponentially gets worse.

It kind of makes match_object unusable at scale, and something to be reserved for very specific use cases.

Maybe theres a way to make it yield or return inconsistent results (if deletes going on, it could return a hole that itl discard for example)?


ets:match_object does yield and return (potentially) inconsistent results.

But maybe it doesn't yield frequent enough.
Have you tried match_object/3 and limit number of returned objects per call?

/Sverker


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200221/e0351712/attachment.htm>