[erlang-questions] JFDB / Erlang Database

[Lyn Headley]

After reading this a little more:

Can you give an example of a range query?
Can you describe the fault-tolerance characteristics?

On Fri, Jul 22, 2016 at 2:59 PM, Lyn Headley <lheadley@REDACTED> wrote:
> This looks very nice to me, and I'm excited to see where it goes. I've
> also had to think a lot lately about the persistence solutions
> available to erlang programmers, and still feel like I haven't fallen
> in love yet. Your list of issues rings quite true to me. My current
> project rolls its own (non-atomic) thing that I'm hoping will be good
> enough, but deep down I can see the problems with it.
>
> Looking forward to seeing this project grow!
>
>
> On Fri, Jul 22, 2016 at 12:37 PM, Jared Flatow <jflatow@REDACTED> wrote:
>> Hi all,
>>
>> A few months ago, I was once again thinking about some of the problems I hit
>> with every new Erlang project. Here’s how it usually goes:
>>
>>  - I want to store data from Erlang, persisted on disk
>>  - I want to be able to search the data using arbitrary indices
>>  - DETS can work, but:
>>    - there’s the size limit
>>    - you have to do a lot of work to cache in memory, and even then it’s all
>> or nothing
>>    - the keys can often take up a lot of (redundant) space, especially for
>> ‘nested’ data
>>    - it doesn’t support any kind of range searches
>>    - writing the data and then the indices is not atomic, so you have to do
>> a lot of work to make sure it’s always consistent
>>  - KV stores have the same problem about managing updating of data / indices
>>  - Relational databases do not let you store / index complex Erlang terms
>> easily
>>  - Sophisticated searching (i.e. logical combinations of keys) is almost
>> non-existent
>>  - Nothing *just works*, persisting and querying data from Erlang requires a
>> ton of thought
>>
>> I decided (finally) to do something about it - my solution is called JFDB
>> (https://github.com/jflatow/jfdb). I’ve been using it for a few months with
>> great success. I thought others here might find it useful, especially with
>> all the recent discussion about new mnesia backends and other Erlang data
>> structures. The implementation is pretty wild - I don’t really know of
>> anything similar - I think the best way to describe it is as a
>> log-structured merge trie.
>>
>> If you try it, I apologize in advance for a lack of documentation, tests,
>> and the inevitable bugs you will hit. I think it’s in pretty good shape now,
>> there are no known bugs, but it of course comes with a big disclaimer about
>> being new and experimental. I’m actively working to improve all these
>> things. If you have any feedback, I would love to hear it!
>>
>> Best,
>> jared
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>