R14B, ETS, read_concurrency

[Rickard Green]

Ryan Zezeski wrote:
> After glancing at the R14B release notes I noticed the SMP related
> improvements that were made.  I specifically find the new read_concurrency
> ETS opt interesting.  I currently have an app that performs roughly 480
> million lookups (in worst case) on multiple ETS tables, with some tables
> having up to 10 million entries.  I perform these lookups in parallel using
> 8 processes (on a 16 core machine).  This is the longest running part of my
> program with a runtime of > 3min.
> 
> My point--I am doing a _huge_ amount of concurrent reads and every
> microsecond counts.  I am curious if anyone could throw a minimum % increase
> of lookups per second I might expect?
> 
> -Ryan

What the effect will be on your application is impossible to guess. This 
depends on how writes are done in the tables using the read_concurrency 
option (if you have lots of write locked operations too you may even get 
a performance degradation), what else your processes do (also other 
processes not accessing the tables), what hardware you use, etc...

In order to give some figures, though. When doing lots of ets lookups 
(only lookups) of small records on a public table from lots (hundreds) 
of processes I get a speedup of about 5 using read_concurrency compared 
to not using it on a dual quad-core machine we got, and about 2.5 on a 
quad-core machine we got. The only thing these processes do are repeated 
ets lookups. You should, however, not expect figures like these for 
normal apps, since they typically do other stuff than ets lookup too.

The only way to know what effect it will have on your app is to try it out.

Regards,
Rickard Green
-- 
Rickard Green, Erlang/OTP, Ericsson AB.