early warning - new rdbms

[Ulf Wiger (AL/EAB)]

Mickael Remond wrote:
> 
> > One interesting experiment might be to define
> > an ISAM table type for really large ordered
> > sets on disk. Combining it with rdbms, you can 
> > type- specify the data types and then convert
> > to whatever format is expected by the ISAM files.
> 
> Could you elaborate a little on this. I was running
> a study to possibly revive the Berkeley DB backend
> idea. My goal is to be able to store huge amount
> of data efficiently with a correct query time in
> Mnesia.

The low-level interface that I've proposed mirrors
the mnesia_lib:db_xxxx() functions (db_put, db_get,
match_object, select, etc.) + some callbacks to
verify cstructs and loading the actual table.

For example, from my first version of a disk_log
table:

db_get(Tab, Key) ->
  Log = log_name(Tab),
  fold_log(
    Log, fun(Terms, Acc) ->
               {continue, 
                lists:foldr(
                  fun(Obj, Acc1)
                    when element(2,Obj) == Key ->
                        [Obj|Acc1];
                    (_Obj, Acc1) ->
                        Acc1
                  end, Acc, Terms)}
   end, []).
                          


db_put(Tab, Val) ->
    disk_log:log(log_name(Tab), Val).

match_object(Tab, Pat) ->
    select(Tab, [{Pat,[],['$_']}]).

select(Tab, Ms) ->
  Log = log_name(Tab),
  MsC = ets:match_spec_compile(Ms),
  RevResult =
    fold_log(
      Log, fun(Terms, Acc) ->
               Matches = ets:match_spec_run(Terms, MsC),
               {continue, lists:reverse(Matches) ++ Acc}
           end, []),
  lists:reverse(RevResult).


(Part of the idea being that you'd mostly do
put() on the log table, and so reading and 
matching need not be super efficient.)

When creating the table, you specify
{external_copies, [{CallBack, Nodes}]}

e.g.

{external_copies, [{rdbms_log, [node()]}]}

and this is understood by mnesia to mean that
the callback module rdbms_log should be used
for the actual reading and writing in this 
table.

For something like Berkeley DB, you'd want to
convert the data to some efficient external 
format. This is helped by rdbms, in the sense
that you can type-specify each attribute. You
can, for example specify that the key _must_
be an integer in the range 0..4294967295.
Then, you can safely encode the key as a 32-bit
unsigned before writing the data to the backend.

The actual code that's generated to verify
the type and range looks something like this,
and doesn't take many microseconds to execute:

validate_person({person,V_id,V_name,V_addr,V_age}) ->
    if
        V_id =/= undefined ->
            true;
        true ->
            throw({type_error,id,{expected,not_null}})
    end,
    if
        is_integer(V_age) ->
            true;
        true ->
            throw({type_error,age,{expected,integer}})
    end,
    validate_oid(V_id,id,oid),
    validate_string(V_name,name,string),
    if
        0 =< V_age, V_age =< 100 ->
            true;
        true ->
            throw({type_error,age,{expected,{inclusive,{0,100}}}})
    end,
    true;

('oid' is a predefined data type in rdbms that 
corresponds to {node(), erlang:now()}, intended
for globally unique identifiers.)

What you do to activate the integrity checking in
rdbms, is simply to e.g. start erlang with 

  erl ... -mnesia access_module rdbms

One caveat: You have to use the mnesia:activity()
functions (of course, without overriding the default
access module with your own choice). The old 
mnesia:transaction/1 function doesn't take access
modules into account.

/Uffe