mnesia and binary/large files

[Klacke]

 > i just read some info about mnesia and the literature suggests that it
 > might not be appropriate for text and binary data.  what would happen if
 > one were to store large text and/or binary datat in a mnesia database and
 > try to use its replication features?



 > i just read some info about mnesia and the literature suggests that it
 > might not be appropriate for text and binary data.  what would happen if
 > one were to store large text and/or binary datat in a mnesia database and
 > try to use its replication features?
 > 

Mnesia has some really cool characteristics and some not so cool.

One of the things mnesia handles poorly is large amounts
of data. Two reasons:

1. The file-storage is performed by dets.erl. dets uses a mark in
each datafile to determine whether the datafile was properly closed 
or not. A datafile is not properly closed if the erlang system crashes,
the host/os crashes (Or erlang is brutally killed by means of
either ^C or halt())
Whenever the datafile is reopened, dets notices this and starts
to repair the datafile. Some  hash structures as well as the freelist
for the file is lost. This just exactly what we need and want unless the
datafile is very very big. A couple of megabyte of disk data takes 
just a couple of seconds to fix, but if the amount is very large
(and databases often are) it can take a considerable amount
of time to fix.

In a production environment, this happens when Erlang or the
host/os crash. Neither of these events happen often.

A possible fix to this problem is for mnesia to use the 
safedets.erl code I just recently posted to www.erlang.org

2. Replication, A replicated table is either loaded from
disk or from the network. No incremental network loads whatsoever
are implemented. A full network load of a very large table can also take
some considerable amount of time.

An obvious mnesia enhancemenet would be the following:
Two nodes A + B, one table T.
A dies, X and Y is inserted in T (on B only), A recovers.
Now instead of copying the entire table T, we only need to
redo the insertions of X and Y at A.

Once I started to implement this but got severe headaches. It's
a lot harder than it sounds.

However, with the safedets fix we should at least be able to
have large non-replicated tables.

As for the issue of binary data, no problems at all. Just
store Erlang binary objects at will, no problem.


Cheers

/klacke

Ps. Now don't go and get all worked up about this, do some
measurements instead and see how long it takes to repair
as well as network load tables of different sizes.
Usually it's ok.


Claes Wikstrom                          Tel: +46 (0)8 692 22 09
Bluetail AB                             Email: klacke@REDACTED
Hantverkargatan 78                      WWW: http://www.bluetail.com
SE-112 38 Stockholm, SWEDEN