[erlang-questions] discussion: mnesia table-specific options

[Morten Krogh]

Hi

It sounds like very good work to make it easy to use all kinds of 
backends with mnesia.

I don't see that ram_copies, disk_copies, disk_only_copies is at a 
differernt level than the backend.

Some backends, as you say, can only operate on disk or in memory. (TC 
can be used in pure memory mode, I would claim, but that is a 
digression. TCMAP in tcutil.c).

That will make the interface a bit strange.

What about this? You specify a list [{node, backend, optional options}]

So if I wanted two nodes node1, node2,

I could write

[{node1, ets}, {node1, tc, options}, {node2, dets}, {node2, mysql, 
options}]

And a very clear set of requirments could be made for new backends, e.g, 
they must present a get, put, erase etc. Then people could even create 
their own
backend very easily. A proplist or gb_tree could become a backend. A 
simple file storage could be a backend.

I could just write

module my_files_module

put(Key, Value) ->
      file:write(Key, term_to_binary(Value).

and similarly for get and erase,

and then plug it into mnesia

[{node1, my_files_module, Dir1}, {node2, my_files_module, Dir2}]

and then there would be a transactional way of saving files on two nodes 
using mnesia.

The choice of memory or disk would then be part of the backend and its 
options, not a separate level. Actually mnesia might not even understand 
what the backend is doing.
The backend could be a remote database, it could be a disk/ram hybrid.

Furthermore, what one could have was a write_only option for the 
backend. Then mnesia would only use put and erase for that backend and 
never issue a get. An append only log file could then be plugged in 
easily as a backend. It would just implement put as file append of {put, 
Key, Value}, and erase as an append of {erase, Key, Value}. But you 
could never query it, except after a crash which would be a special case.

For crash recovery the backends could present an iterator through all 
values.

Cheers,

Morten.





On 12/20/10 2:56 PM, Ulf Wiger wrote:
> Given that there are now several interesting performance options
> for ets, and a 64-bit dets version is (sort of) in the works, it seems
> a good time to consider how these things can be reflected in
> mnesia table definitions.
>
> Some time ago, I introduced an 'external_copies' type in mnesia,
> and this was used (with some modifications) by mnesiaex to provide
> a TokyoCabinet back-end to mnesia. Thesis projects at Klarna have
> played around with CouchDB backends etc.
>
> I think that conceptually, it would seem good to keep the
> ram_copies, disc_copies and disc_only_copies, regardless of
> back-end, since they address higher-level access characteristics
> (e.g. TokyoCabinet is, strictly speaking, also disc_only.)
>
> A form of behaviour option could then be added that gives
> additional options - e.g. tuning parameters to InnoDB, dets, ets,
> or whatever back-end is being used.
>
> Taking it one step further, it should be possible to specify a
> default behaviour for each copy type, and override per-table.
>
> Comments?
>
> BR,
> Ulf W
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>