locked up system using :ets.match_object

[Sverker Eriksson]

On fre, 2020-01-17 at 20:09 +0200, Led wrote:
> > I am having some performance trouble in a system that does a few queries on
> > a small ets table of around 10,000 records.
> > 
> > Basically with around 500 concurrent processes, everything is fine, 1500 I
> > start to notice some small degradation, at around 3000 concurrent processes
> > the schedulers grind to a halt, TOP system CPU usage is around 50%, but
> > Erlang scheduler usage (scheduler:utilization) is 100% and capped out on all
> > 40 threads.
> > 
> > I am guessing the schedulers are all waiting on locks on the ets table.  I
> > thought match_object and ets was quite optimized these days, using R22, I am
> > wondering if there is some synchronization/locking issues that could be
> > addressed.  Because I mean at 3000 processes maybe hitting that table 10
> > times per second on average, does not seem like much. 30k match_objects per
> > second, with ongoing inserts. 
> > 
> > Also would there be a way to debug/pinpoint this is the exact issue?  I just
> > did A/B testing where I turned off parts of the system, when I turned off
> > the part that does the match_objects on the ETS table, the system ran fine
> > and never deadlocked at 100% scheduler usage.  Its also hard to profile, as
> > the system is so locked up the profiler barely runs.
> > 
> > For now it seems the solution is to rework the architecture and put a second
> > cached view ETS table, so the match_objects can be replaced with key
> > lookups.  Which gets filled by a single process running that pulls via
> > match_object from the main table and fills the cache.
> > 
> You didn't specify parameters of your table.
> 
> 
And what's the frequency of those inserts that you mention.
ets:match_object is a read-only operation and should only inflict lock
contention with other write operations, such as ets:insert.
/Sverker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200117/0ad0c39d/attachment.htm>