<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>On fre, 2020-01-17 at 20:09 +0200, Led wrote:</div><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr"></div><blockquote type="cite"><div><div style="font-family:Helvetica Neue,Helvetica,Arial,sans-serif;font-size:13px"><div dir="ltr">I am having some performance trouble in a system that does a few queries on a small ets table of around 10,000 records.<br><br>Basically with around 500 concurrent processes, everything is fine, 1500 I start to notice some small degradation, at around 3000 concurrent processes the schedulers grind to a halt, TOP system CPU usage is around 50%, but Erlang scheduler usage (scheduler:<span>utilization</span>) is 100% and capped out on all 40 threads.<br><br>I am guessing the schedulers are all waiting on locks on the ets table. I thought match_object and ets was quite optimized these days, using R22, I am wondering if there is some synchronization/locking issues that could be addressed. Because I mean at 3000 processes maybe hitting that table 10 times per second on average, does not seem like much. 30k match_objects per second, with ongoing inserts. <br><br>Also would there be a way to debug/pinpoint this is the exact issue? I just did A/B testing where I turned off parts of the system, when I turned off the part that does the match_objects on the ETS table, the system ran fine and never deadlocked at 100% scheduler usage. Its also hard to profile, as the system is so locked up the profiler barely runs.<br><br>For now it seems the solution is to rework the architecture and put a second cached view ETS table, so the match_objects can be replaced with key lookups. Which gets filled by a single process running that pulls via match_object from the main table and fills the cache.</div></div></div><br></blockquote></div><div><br></div><div></div><span class="gmail-tlid-translation gmail-translation" lang="en"><span title="" class="gmail-">You didn't specify parameters of your table.</span></span><br><div><br></div><div><br></div></div></blockquote><div><br></div><div>And what's the frequency of those inserts that you mention.</div><div><br></div><div>ets:match_object is a read-only operation and should only inflict lock contention with other write operations, such as ets:insert.</div><div><br></div><div><br></div><div>/Sverker</div><div><br></div></body></html>