[erlang-questions] JFDB / Erlang Database

[Jared Flatow]

Thanks for your enthusiasm :)

As far as range queries - these are currently only exposed through the Erlang API via the prefix queries. However, an arbitrary {StartKey, StopKey} could easily be added to the jfdb:fold/3 function if needed. I just haven’t really needed anything beyond prefixes yet.

As far as fault tolerance, the DB is designed to be incorruptible no matter if/when it crashes. Changes are appended to the keys file, until compaction, at which point a new keys file with a single level is created, and then atomically replaces the old one. In general if you don’t flush, it’s possible changes have not hit the disk yet. Whenever you want to be sure that something hits disk, you can flush. Space in the vals file is never reused until one commit (to disk) after the deallocation occurs in the keys file.

> On Jul 22, 2016, at 4:41 PM, Lyn Headley <lheadley@REDACTED> wrote:
> 
> After reading this a little more:
> 
> Can you give an example of a range query?
> Can you describe the fault-tolerance characteristics?
> 
> On Fri, Jul 22, 2016 at 2:59 PM, Lyn Headley <lheadley@REDACTED> wrote:
>> This looks very nice to me, and I'm excited to see where it goes. I've
>> also had to think a lot lately about the persistence solutions
>> available to erlang programmers, and still feel like I haven't fallen
>> in love yet. Your list of issues rings quite true to me. My current
>> project rolls its own (non-atomic) thing that I'm hoping will be good
>> enough, but deep down I can see the problems with it.
>> 
>> Looking forward to seeing this project grow!
>> 
>> 
>> On Fri, Jul 22, 2016 at 12:37 PM, Jared Flatow <jflatow@REDACTED> wrote:
>>> Hi all,
>>> 
>>> A few months ago, I was once again thinking about some of the problems I hit
>>> with every new Erlang project. Here’s how it usually goes:
>>> 
>>> - I want to store data from Erlang, persisted on disk
>>> - I want to be able to search the data using arbitrary indices
>>> - DETS can work, but:
>>>   - there’s the size limit
>>>   - you have to do a lot of work to cache in memory, and even then it’s all
>>> or nothing
>>>   - the keys can often take up a lot of (redundant) space, especially for
>>> ‘nested’ data
>>>   - it doesn’t support any kind of range searches
>>>   - writing the data and then the indices is not atomic, so you have to do
>>> a lot of work to make sure it’s always consistent
>>> - KV stores have the same problem about managing updating of data / indices
>>> - Relational databases do not let you store / index complex Erlang terms
>>> easily
>>> - Sophisticated searching (i.e. logical combinations of keys) is almost
>>> non-existent
>>> - Nothing *just works*, persisting and querying data from Erlang requires a
>>> ton of thought
>>> 
>>> I decided (finally) to do something about it - my solution is called JFDB
>>> (https://github.com/jflatow/jfdb). I’ve been using it for a few months with
>>> great success. I thought others here might find it useful, especially with
>>> all the recent discussion about new mnesia backends and other Erlang data
>>> structures. The implementation is pretty wild - I don’t really know of
>>> anything similar - I think the best way to describe it is as a
>>> log-structured merge trie.
>>> 
>>> If you try it, I apologize in advance for a lack of documentation, tests,
>>> and the inevitable bugs you will hit. I think it’s in pretty good shape now,
>>> there are no known bugs, but it of course comes with a big disclaimer about
>>> being new and experimental. I’m actively working to improve all these
>>> things. If you have any feedback, I would love to hear it!
>>> 
>>> Best,
>>> jared
>>> 
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>