[erlang-questions] data managment layer on top of K-V stores

[Ulf Wiger]

Hi Joe,

We are working on something along these lines. It works pretty well already, but documenting, slapping licenses and releasing to the public is not our priority right now (although all those things are planned for TNF [1]). We most definitely intend to Open-Source our generic components.

Our 'kvdb' library currently has these features:

- supports two different backends: leveldb and sqlite3
- basic KV store API (put, get, delete, prefix_match, ets-style select)
- queue-type tables supporting push and pop (fifo or lifo)
- schema validation, although it's mainly empty hooks today

So far, we're using it in-house to do some nifty storing of configuration trees, where we can cheaply skip over sub-trees without hurting leaf-level performance. Future XPATH-style queries would be handy, but right now, we can live without them.

My own plan is to add a transaction layer on top, using my 'ddd' [2] library, also to be released in TNF. But I'll have to prioritize it against other work, of course.

As for atomic sequences, we pipe all updates through a gen_server, although you can fetch the database handle and bypass it at your own risk. We have a means to start multiple session servers, but not yet any arbitration protocol between them.

BR,
Ulf W

[1] TNF = The Near Future (™) = some as-yet-undefined point in future time
[2] 'ddd' = Distributed Deadlock Detection - a protocol-based scalable deadlock-free algorithm whose time, I think, has come. I've had it in my back pocket since 1993. Thomas Arts verified it for me, but I've never had opportunity or reason to finish the job.

On 29 Feb 2012, at 14:40, Joe Armstrong wrote:

> Has anybody made (in Erlang) a database managing layer that sits on top of
> a Key-Value store.
> 
> What I imagine is the following:
> 
>    1) at the bottom there is a K-V store  with a primitive set of operations
>        get(key) put(key,Val) foldkeys(F, Acc) foldKV(F, Acc)
> 
>    2) At the top level - I have typed record (say)
> 
>      -record(person, {name :: binary(), arg:: integer, ...})
> 
>      I'd like the say
> 
>            db:store(#person{...})
> 
>      To store a new tuple in the database - this would involve a few
> locked KV operations on the
>       underlying KV store. I'd have to fetch a schema, type check the
> arguments, made a new index and so
> 
>       I'd like to also search the database
> 
>          db:search([{record,person},{name,{matchesRegExp,...}])
> 
>       and so on
> 
>   There seem to be a lot of alternative for the bottom layer - but
> what about the glue between the bottom layer
>    and the top level - has anybody done any work on this? (I guess a
> lot of mnesia does this - since the bottom layers
>    are just ets and dets) but I'd like to play with other bottom layers.
> 
>   I  also need to lock the KV store for short times - instead of the
> regular get and put I feel I'd like a
> function
> 
>         withKeys( [ Key ], fun doit/1)
> 
>        which looks up Key1, Key2, .. in [ Keys] and assumes this
> returns a list L = [{K1, {yes, V1} }, {K2, no} etc.
>        it then calls doit(L) - doit(L) must return {Val, [{K,V}]}
> 
>        Val is the return value and the list [{K,V}] is injected back
> into the store.
> 
>         This is a simple form of atomic transaction which assumes
> that all the write are performed only if doit succeeds
>         otherwise the store is not changed.
> 
>     Pure K-V stores seem to need a little bit of extra glue on top -
> some kind of atomic transactions and type checking and
>     schema control - mnesia does all of this - but the glue seems to
> be inseparable so I can't use it with a different backend.
> 
> 
>   /Joe
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions