[erlang-questions] discussion: mnesia table-specific options

[Ulf Wiger]

On 20 Dec 2010, at 16:56, Morten Krogh wrote:

> Hi
> 
> It sounds like very good work to make it easy to use all kinds of backends with mnesia.
> 
> I don't see that ram_copies, disk_copies, disk_only_copies is at a differernt level than the backend.
> 
> Some backends, as you say, can only operate on disk or in memory. (TC can be used in pure memory mode, I would claim, but that is a digression. TCMAP in tcutil.c).

You're right, of course, but I maintain that it can be a useful 
distinction to keep the ram/disc/disc_only types as defined in mnesia,

The most important distinction between RAM and DISK storage is 
that RAM-only storage (ram_copies) is not persistent, i.e. does not 
survive a system restart. It is also expected to be fast, but to me, that's 
a secondary consideration.

The special RAM+DISK combo (disc_copies) combines persistency with 
fast lookup, but is, as a consequence, limited by available RAM.
In the earliest versions of mnesia, the disk part was handled by dets tables,
but dets was later replaced by disk_log. The only visible difference to the user
was that log dumps became much faster.

DISK-only may (and usually does) employ any form of smart caching, but
is not expected to be limited by available RAM. Any number of good 
backends could be used instead of dets here, perfectly transparently to 
the user.

Your suggestion (below) was more or less how external_copies worked,
although the options for the backend were passed as user properties,
which was something of a kludge.

BR,
Ulf

> That will make the interface a bit strange.
> 
> What about this? You specify a list [{node, backend, optional options}]
> 
> So if I wanted two nodes node1, node2,
> 
> I could write
> 
> [{node1, ets}, {node1, tc, options}, {node2, dets}, {node2, mysql, options}]
> 
> And a very clear set of requirments could be made for new backends, e.g, they must present a get, put, erase etc. Then people could even create their own
> backend very easily. A proplist or gb_tree could become a backend. A simple file storage could be a backend.
> 
> I could just write
> 
> module my_files_module
> 
> put(Key, Value) ->
>     file:write(Key, term_to_binary(Value).
> 
> and similarly for get and erase,
> 
> and then plug it into mnesia
> 
> [{node1, my_files_module, Dir1}, {node2, my_files_module, Dir2}]
> 
> and then there would be a transactional way of saving files on two nodes using mnesia.
> 
> The choice of memory or disk would then be part of the backend and its options, not a separate level. Actually mnesia might not even understand what the backend is doing.
> The backend could be a remote database, it could be a disk/ram hybrid.
> 
> Furthermore, what one could have was a write_only option for the backend. Then mnesia would only use put and erase for that backend and never issue a get. An append only log file could then be plugged in easily as a backend. It would just implement put as file append of {put, Key, Value}, and erase as an append of {erase, Key, Value}. But you could never query it, except after a crash which would be a special case.
> 
> For crash recovery the backends could present an iterator through all values.
> 
> Cheers,
> 
> Morten.
> 
> 
> 
> 
> 
> On 12/20/10 2:56 PM, Ulf Wiger wrote:
>> Given that there are now several interesting performance options
>> for ets, and a 64-bit dets version is (sort of) in the works, it seems
>> a good time to consider how these things can be reflected in
>> mnesia table definitions.
>> 
>> Some time ago, I introduced an 'external_copies' type in mnesia,
>> and this was used (with some modifications) by mnesiaex to provide
>> a TokyoCabinet back-end to mnesia. Thesis projects at Klarna have
>> played around with CouchDB backends etc.
>> 
>> I think that conceptually, it would seem good to keep the
>> ram_copies, disc_copies and disc_only_copies, regardless of
>> back-end, since they address higher-level access characteristics
>> (e.g. TokyoCabinet is, strictly speaking, also disc_only.)
>> 
>> A form of behaviour option could then be added that gives
>> additional options - e.g. tuning parameters to InnoDB, dets, ets,
>> or whatever back-end is being used.
>> 
>> Taking it one step further, it should be possible to specify a
>> default behaviour for each copy type, and override per-table.
>> 
>> Comments?
>> 
>> BR,
>> Ulf W
>> 
>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>> http://erlang-solutions.com
>> 
>> 
>> 
>> 
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> 
> 
> 
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> 

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com