[erlang-questions] Surprising overhead to a Mnesia transaction:

[jim rosenblum]

Dear List

I need some Mnesia help, please. I will try to be precise in my question
and give appropriate context: I am really grateful to all of you who help
the rest of us cultivate some skill with Erlang.

I developed a special-purpose, in-memory distributed cache for our
company's Java application. It has been very successful thus far. My need
was for a cache consisting of about 10,000 rows that needed to be
distributed amongst 2 - 4 nodes and be capable of handling 100s of
transactions per second at most. All operations (put, evict, etc.) can
happen at any node.

My current version has been in production, doing fine. According to
basho_bench (admittedly synthetic), the cache is good to about 20,000 ops
per second with mean latency for puts at 0.75 ms at the 95th percentile.

And, up until now, all transactions have been in a sync-dirty context.

As is typical in these types of environments we have a race-condition that
we are trying to solve. Java client, A, does a put:(K, V0) to the cache
while, at about the same time, another Java client, B, puts an updated
version of the same key - put:(K,V1). And, of course, it can happen that
the second write gets presented to the cache first putting the "newest"
value, V1, in the cache and then the first write gets presented to the
cache putting the "stale" V0 in the cache.

Our strategy for addressing this is that the Java application will include
a strictly monotonic sequence number with each Put. The cache will maintain
a table that persists the highest sequence number that it has seen and if a
Put with a smaller sequence number is presented to the cache, the cache
rejects the Put; otherwise it writes the new sequence to the sequence table
and does the Put.

My implementation of the above involves having a replicated table which
persists the highest sequence number seen, and I wrap the following steps
in an async_transaction
1. Compare the sequence number in the Put with the "high-water-mark"
sequence number
2. If the Put's sequence number is higher, do the put and update the
high-water-mark;
3. else, abandon the put

I knew that there would be a cost to a non-dirty transaction, but I was
really surprised at the magnitude of the cost. Puts went from
20,000 ops per second with mean latency at 0.75 ms to 1,500 ops per second
with mean latency of 12ms.

So, now to my questions:
1. If I am trying to transactionally protect this sequence table, do I need
to use sync_transaction or will async_transaction work. I don't understand
what an ACID transaction means when there are more than 1 node and a
transaction context which is async. If two Puts are happening on two
different nodes, will one block until the other completes EVEN when in
async mode?
2. is the magnitude of the performance hit seem "normal"?
3. Is there a better way to achieve what I am looking for then the above?


Thanks very much... and some code if it helps...


put(Key, Value, SeqNo) ->
    F = fun() ->
case mnesia:wread(seq, global) of
   [] ->
simple_put(Key, Value),
mnesia:write(#seq{key=global, value = SeqNo});
   [#seq{value = Old}=S] when SeqNo > Old ->
simple_put(Key, Value),
mnesia:write(S#seq{value = SeqNo});
   [#seq{value = Old}] ->
lager:warning("~p: out of order put. Map: ~p, Old: ~p, New: ~p.",
     [?MODULE, Old, SeqNo]),
{out_of_seq, {put, Key, Old, SeqNo}}
end
end,
    case mnesia:sync_transaction(F) of
{atomic, {out_of_seq, _}} = R ->
   R;
{atomic, Result} ->
   Result;
{aborted, Reason} ->
   {error, Reason}
    end.




e
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150821/c49b8e57/attachment.htm>