[erlang-questions] 20k messages in 4s but want to go faster!

[Richard Carlsson]

Joel Reymont wrote:
> On Jul 10, 2009, at 8:45 PM, Matthew Sackman wrote:
> 
>> Also, I have to say, I wouldn't use a dict. Immutable data structures in
>> Erlang are _slow_ when compared to, eg, ets.

It depends on the data and what you need to do with it. If each entry
is rather large and you do a lot of retreival and/or insertion, you will
spend a whole lot of time copying data between the process heap and the
table (and cause more garbage collection as a consequence). An immutable
data structure resides on the process heap all the time. Of course, if
the table size gets huge enough, the log(n) factor will eventually be
greater than the work spent in copying; as usual, you'll simply have
to measure.

That said, ETS tables have some other advantages as well, arguably the
most important being that you can use fast match specifications and
qlc for searching. If you go with ETS tables, you may want to think
about normalizing your representation and splitting it over several
tables, to keep the size of each table row as small as possible. (But
if you need to read every part of an entry each time anyway, this does
not help.)

    /Richard