[erlang-questions] mnesia vs ets

[Roberto Ostinelli]

> dirty_read adds 2 other things (two additional ets lookups) where
> is table located (which node) and the stoarge type, ets or dets.
>
> So dirty_read abstracts away the location and storage type which you
> implicit knows
> when calling ets:lookup/2.

thank you dan, that clears it up.


2009/10/28 Ulf Wiger <ulf.wiger@REDACTED>:
> In some cases, this flexibility is not worth as much as the raw
> speed. Personally, I always start out using transactions, and
> always using mnesia:activity(Type, F). I have very rarely had to
> conclude that transactions weren't fast enough for my needs.
>
> BTW, dirty reads to _not_ support fragmented tables out of the
> box. Neither does the standard mnesia:transaction(F).
> You need to use mnesia:activity(Type, F, mnesia_frag).

hi ulf,

i do not get lost in premature optimization :) i have clear specifics
that state a need for me to read 10,000 key/value pairs/sec.
therefore, i'm benchmarking to see various speeds, so that i can then
decide for the best speed/distribution/transaction tradeoff.

i thought i might share the results with you:


1. mnesia table in 6 fragments distributed over 2 erlang nodes, created with:

FragProps = [{node_pool, ['1@REDACTED', '2@REDACTED']}, {n_fragments,
6}, {n_disc_copies, 1}],
mnesia:create_table(?TABLENAME, [{frag_properties, FragProps},
{attributes, record_info(fields, test)}]).

- write [speed: 370.19 rows/min]:

WriteFun = fun() -> [mnesia:dirty_write({?TABLENAME, K, {}}) || K <-
lists:seq(1, Num)] end,
mnesia:activity(sync_dirty, WriteFun, mnesia_frag).

- read with transaction [speed: 361.10 rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(transaction, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].

- read dirty [speed: 552.25 rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(sync_dirty, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].


2. mnesia table in 6 fragments over a single erlang node, created with:

FragProps = [{node_pool, ['1@REDACTED']}, {n_fragments, 6}, {n_disc_copies, 1}],
mnesia:create_table(?TABLENAME, [{frag_properties, FragProps},
{attributes, record_info(fields, test)}]).

- write [speed: 3451.64 rows/min]:

WriteFun = fun() -> [mnesia:dirty_write({?TABLENAME, K, {}}) || K <-
lists:seq(1, Num)] end,
mnesia:activity(sync_dirty, WriteFun).

- read with transaction [speed: 947.47 rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(transaction, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].

- read dirty [speed: 10900.37 rows/min rows/min]:

ReadFun = fun(Key) -> mnesia:read({?TABLENAME, Key}) end,
[mnesia:activity(sync_dirty, ReadFun, [K], mnesia_frag) || K <-
lists:seq(1, ?NUM)].


3. mnesia table no fragments on single node, created with:

mnesia:create_table(?TABLENAME, [{disc_copies, ['1@REDACTED',
'2@REDACTED']}, {attributes, record_info(fields, test)}]).

- write [speed: 3545.29 rows/min]:

WriteFun = fun() -> [mnesia:dirty_write({?TABLENAME, K, {}}) || K <-
lists:seq(1, Num)] end,
mnesia:activity(sync_dirty, WriteFun).

- read with transaction [speed: 2148.44 rows/min]:

[mnesia:transaction(fun(K) -> mnesia:read({?TABLENAME, K}) end) || K
<- lists:seq(1, ?NUM)].

- read dirty [speed: 29384.40 rows/min]:

[mnesia:dirty_read(?TABLENAME, K) || K <- lists:seq(1, ?NUM)].


4. mnesia table with no fragments on single node.

- read [speed: 91975.16 rows/min]


interestingly enough:

a. write on mnesia table no fragments on single node is same speed
than with fragments.
b. read [transaction/dirty] on mnesia table without fragments is 3x
faster than with fragments.
c. using ets to read a disc_copy not fragmented mnesia table is 3x
faster than using mnesia dirty_write.


does this meet your expectations? any comments always welcome :)

r.