[erlang-questions] mnesia_leveldb + prefix keys

[Mikael Pettersson]

On Tue, Jan 22, 2019 at 11:32 AM Matthias Rieber <ml-erlang@REDACTED> wrote:
>
> Hello,
>
> I've a question regarding the prefix keys in mnesia_eleveldb. I have a
> record like this:
>
> -record(key, {
>   jid       :: { binary(), binary() },
>   timestamp :: erlang:timestamp()
> }).
>
> -record(table, {
>   key = key{},
>   payload :: term()
> }).
>
>
> As far as I understand these queries are fast:
>
> mnesia:match_object(
>   #table{key=#key{jid={<<"user1">>, <<"localhost">>},
>                   timestamp = '_'},
>          payload='_'})
>
> Since the keys are ordered I expected that query to be fast:
>
> mnesia:select(table, ets:fun2ms(
>  fun(#table{key=#key{jid={<<"user1">>, <<"localhost">>},
>                      timestamp = Timstamp},
>             payload=P}) when Timstamp > {1,2,3} -> P
>  end)
>
> But performance tests show that this will read all records (probably only
> the ones with the matching jid). Is it possible to select this in an
> efficient manner?

Your observations are correct.  The second case is a "range select"
which LevelDB can support efficiently, but mnesia_eleveldb doesn't
implement that optimization yet.  I've had a ticket to implement this
for a while now, but the initial prototype didn't work and I haven't
had time to up-prioritize it yet.

Meanwhile we've implemented another optimization in mnesia_eleveldb:
we no longer store the tags of records in mnesia_eleveldb tables as
those tags are invariant and redundant.  This reduces I/O and CPU
usage.

/Mikael