<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi all,<div class=""><br class=""></div><div class="">A few months ago, I was once again thinking about some of the problems I hit with every new Erlang project. Here’s how it usually goes:</div><div class=""><br class=""></div><div class=""> - I want to store data from Erlang, persisted on disk</div><div class=""> - I want to be able to search the data using arbitrary indices</div><div class=""> - DETS can work, but:</div><div class=""> - there’s the size limit</div><div class=""> - you have to do a lot of work to cache in memory, and even then it’s all or nothing</div><div class=""> - the keys can often take up a lot of (redundant) space, especially for ‘nested’ data</div><div class=""> - it doesn’t support any kind of range searches</div><div class=""> - writing the data and then the indices is not atomic, so you have to do a lot of work to make sure it’s always consistent</div><div class=""> - KV stores have the same problem about managing updating of data / indices</div><div class=""> - Relational databases do not let you store / index complex Erlang terms easily</div><div class=""> - Sophisticated searching (i.e. logical combinations of keys) is almost non-existent</div><div class=""> - Nothing *just works*, persisting and querying data from Erlang requires a ton of thought</div><div class=""><br class=""></div><div class="">I decided (finally) to do something about it - my solution is called JFDB (<a href="https://github.com/jflatow/jfdb" class="">https://github.com/jflatow/jfdb</a>). I’ve been using it for a few months with great success. I thought others here might find it useful, especially with all the recent discussion about new mnesia backends and other Erlang data structures. The implementation is pretty wild - I don’t really know of anything similar - I think the best way to describe it is as a log-structured merge trie.</div><div class=""><br class=""></div><div class="">If you try it, I apologize in advance for a lack of documentation, tests, and the inevitable bugs you will hit. I think it’s in pretty good shape now, there are no known bugs, but it of course comes with a big disclaimer about being new and experimental. I’m actively working to improve all these things. If you have any feedback, I would love to hear it!</div><div class=""><br class=""></div><div class="">Best,</div><div class="">jared</div></body></html>