[erlang-questions] "Erlang plus BDB: Disrupting the Conventional Web Wisdom"

[Chris Newcombe]

> how much more abstraction is required to bring the BDB API up to the level
> of RDBMS?

BDB keys and records are just arbitrary byte sequences, so you can do
whatever you like.

It's trivial (and fast) to use term_to_binary and binary_to_term to
encode arbitrary records.  (I'd recommend not using that for keys
though -- would be bad if Ericsson changed the external term format
between releases.)

Caveat: encoding 'real Erlang records' this way will break if you
change the record definition across software versions and don't
upgrade the data simultaneously.  To avoid that, either add a version
number member to all records plus code to read them from BDB as plain
tuples and auto-upgrade old versions to the proper #record, or, store
data as lists of tagged tuples.

For 'simple' cases, I do the latter, with very short tags to avoid
wasting space, and macros in the code that map to the tag names, so
that the code remains readable:

    [{n, Name}, {a, Address}]

in code would appear as:

    [{?USER_NAME_T, Name}, {?USER_ADDRESS_T, Address}]

For more complex stuff (or external compatibility) use XML, or JSON.
You could store them textually, or after being parsed into Erlang
terms.

If your application has a few hard-wired data-access patterns, you
just code for those directly.  If you want to build little custom
query languages you can do that too.  e.g. I have a friend who wrote a
prolog interpretter in Erlang and stored the fact-base and rules in
BDB, so you could run prolog queries on it.

Obviously indexing is vital to performande of many applications. But
'indexes' are just other tables that happen to map secondary keys to
primary keys.  You can maintain indexes manually as part of your
transactions (see the examples_*.erl files from the tarball).  Or you
could write a little framework (a layer/API on top of the current
driver) that supported declarative indexes and queries.  There's
infinite literature on query optimizers to look at :)

The BDB product line includes an excellent XML database with full
XQuery support and declarative indexing.  I haven't looked closely at
supporting this via EDTK yet, but it might not be too hard.  The main
problem is that BDB XML is written in C++ and EDTK currently only
supports C libraries.  So EDTK would have to be compiled as C++ (but
still linkable to the Erlang VM which is pure C), in order to catch
exceptions thrown by BDB XML calls.  But the current general framework
should all work just fine.


> I'm not thinking of a SQL frontend to BDB, but more like having a
> easy mechanism ( e.g. single function call) to create tables, select
> records, insert records, etc.

In BDB parlance an RDBMS 'table' is called a 'database'.   The current
EDTK driver makes this a one-liner.  In fact, this is how you access
an existing database tool

    DB = ?BDB_PC:get_db_handle(BdbPort, bdb_DB_BTREE, "test.db"),

> select records,

BDB supports lookup by primary key or (with btree databases)
key-prefix, so this is trivial:

    MyData = binary_to_term( ?BDB:db_get_data(BdbPort, DB, Txn, Key, []) ),

(BTW, BDB also has a FIFO 'queue' database type for fixed-length
records.  The current EDTK driver contains a sketch (not yet
production code, but close) of how to use that to build persistent
transactional message queues.)

> insert records,

Again, trivial (unless you mean with declarative indexing, triggers,
typed tables etc)

    ?BDB:db_put(BdbPort, DB, void, Key, term_to_binary(Data), []).

Or you can just store raw binaries of course :)

Chris