[erlang-questions] [ANN] LETS - LevelDB-based Erlang Term Storage v0.5.3

[Dave Smith]

On Tue, Nov 22, 2011 at 4:30 PM, Ciprian Dorin Craciun
<ciprian.craciun@REDACTED> wrote:
>    * and it gets worse when the key does not exist; (the performance
> drops to about a couple of ops per second...)

Yes, LevelDB is very, very bad when it comes to checking for keys that
do not exist on a big dataset. It's proportional to the number of
levels -- if you have 6 levels you may wind up doing 6 random disk
seeks (as I understand the code). Obviously, caching can help, but if
the access pattern is random enough it'll hurt badly.

I did ask on the LevelDB list about using bloom filters to address
this issue, but there were concerns by the LevelDB team about that
approach:

http://comments.gmane.org/gmane.comp.db.leveldb/249

D.