[erlang-questions] Mnesia

[Ulf Wiger]

> On 14 Oct 2015, at 05:19, Lloyd R. Prentice <lloyd@REDACTED> wrote:
> 
> So if we dig into the code, what exactly needs to change to make that happen?

(ah, I never sent this … I’ll finish the thought, even though others have answered)

Well, to some extent, I think Mnesia is its own worst enemy here.

The wonderful thing about Mnesia is that it acts practically as an extension of the Erlang environment: it runs in the same memory space and can allow access to data in tens of usecs. It also supports both dynamic and transparent handling of data - no schema info to speak of, and ad-hoc replication. While it is possible to use Mnesia in a way that scales pretty well, Mnesia practically begs you not to.

Considering the concrete challenges in Mnesia, I can think of a few off the top of my head:

- Table replication. Mnesia sucks the entire table over every time, which is fine for small tables, but less so for large ones. While networks have gotten much faster, table size seems to increase at an even higher rate.

Mnesia_ext offers a bit of flexibility, by letting each backend plugin determine what should be sent [1]. While a backend could get creative within the bounds of the table synch protocol, one should remember that mnesia doesn’t require backens to be the same on different nodes.

- Table load. This is mainly an issue with dets repair, and the leveldb backend is *much* faster in this regard.

- Split brain. Mnesia offers some low-level support, but additional, non-trivial logic is required (unsplit seems to help). Note that few ACID dbms:es offer much better support, but it arguably hits mnesia harder since it entices people to jump into data replication schemes they would hardly attempt (or even *could* attempt) in e.g. Postgres. Split brain is a hairy issue, and will remain so.

- Sharding. Mnesia’s sharing support (mnesia_frag), while cool, is an afterthought - and it shows. It is a bit too non-transparent, and rehashing and (lack of) ordered_set semantics* are issues. There was some attempt at implementing consistent hashing support on top of mnesia_frag - as I recall, it was semi-successful. In general, for both sharding and netsplits, perhaps mnesia could use a meta layer, but this is hard to add without incuring penalties on use cases where it doesn’t add much utility.

- Indexing. In stock mnesia, indexes are rebuilt on table load. In mnesia_ext, there is support for keeping persistent indexes, and noting when they are consistent with the data so they don’t have to be rebuilt. While not perfect, it is at least an improvement. Indexes have also been improved in a few other ways in mnesia_ext.

- Backup. I’m not sure that mnesia’s backup scheme scales to gigantic data. Klarna has its own system for backup, based on their custom replication protocol.

- Locking. Richard mentioned this. When executing large transactions or running workloads with several competing actors, transaction restarts can become a real issue. I’ve toyed with the idea of trying to insert the ‘locks’ locker in mnesia, but the main problem would likely be that it would incur a noticeable penalty on simple work, which makes it practically a non-starter in a legacy product.

But basically, one should probably think twice before taking an ACID largely-in-memory database and trying to extend it to tackle Big Data. I think mnesia has a sweet spot as a deeply embedded data store - essentially going where no other DBMS can go**. If some of the more debilitating flaws are addressed, I think mnesia can be the best choice for quite a few applications, possibly combined with a more traditional DBMS.

[1] https://github.com/klarna/otp/blob/OTP-18.0-mnesia_ext/lib/mnesia/test/mnesia_ext_filesystem.erl#L152
* Not that sharing ordered_set tables efficiently is particularly easy to begin with.
** One can look at it this way: mnesia is bad at some of the things e.g. riak is good at, but OTOH, riak is terrible for some of the use cases where mnesia shines. Being great at something also means deciding what you’re willing to be bad at, and for mnesia, that decision was made long ago.

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com