[erlang-questions] JFDB / Erlang Database

[Lyn Headley]

This looks very nice to me, and I'm excited to see where it goes. I've
also had to think a lot lately about the persistence solutions
available to erlang programmers, and still feel like I haven't fallen
in love yet. Your list of issues rings quite true to me. My current
project rolls its own (non-atomic) thing that I'm hoping will be good
enough, but deep down I can see the problems with it.

Looking forward to seeing this project grow!


On Fri, Jul 22, 2016 at 12:37 PM, Jared Flatow <jflatow@REDACTED> wrote:
> Hi all,
>
> A few months ago, I was once again thinking about some of the problems I hit
> with every new Erlang project. Here’s how it usually goes:
>
>  - I want to store data from Erlang, persisted on disk
>  - I want to be able to search the data using arbitrary indices
>  - DETS can work, but:
>    - there’s the size limit
>    - you have to do a lot of work to cache in memory, and even then it’s all
> or nothing
>    - the keys can often take up a lot of (redundant) space, especially for
> ‘nested’ data
>    - it doesn’t support any kind of range searches
>    - writing the data and then the indices is not atomic, so you have to do
> a lot of work to make sure it’s always consistent
>  - KV stores have the same problem about managing updating of data / indices
>  - Relational databases do not let you store / index complex Erlang terms
> easily
>  - Sophisticated searching (i.e. logical combinations of keys) is almost
> non-existent
>  - Nothing *just works*, persisting and querying data from Erlang requires a
> ton of thought
>
> I decided (finally) to do something about it - my solution is called JFDB
> (https://github.com/jflatow/jfdb). I’ve been using it for a few months with
> great success. I thought others here might find it useful, especially with
> all the recent discussion about new mnesia backends and other Erlang data
> structures. The implementation is pretty wild - I don’t really know of
> anything similar - I think the best way to describe it is as a
> log-structured merge trie.
>
> If you try it, I apologize in advance for a lack of documentation, tests,
> and the inevitable bugs you will hit. I think it’s in pretty good shape now,
> there are no known bugs, but it of course comes with a big disclaimer about
> being new and experimental. I’m actively working to improve all these
> things. If you have any feedback, I would love to hear it!
>
> Best,
> jared
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>