mnemosyne and BIG databases (Re: mnesia power index -- What about fragmentation?)

[hasse@REDACTED]

[Valentin Micic:]
> On a different note, I've been playing with menmosyne and
> as much as I liked it, I'm not so sure that it is intended
> for BIG databases. Actually, I've seen run-time crashed
> due to the bad/buggy query (it run out of memory). This
> is a bit concerning and preatty un-erlang like. Should one
> encourage usage of mnemosyne on a big databases?

[Ulf Wiger:]
> OTOH, on QLC, the manual states:
> 
> "Support for faster join of two tables will be added not later
> than in R11. Depending on preferences and priorities some high
> level optimizations may be added in the future."
> 
> My interpretation is that QLC can hardly be recommended for
> big databases either - at least not at this time. QLC might
> still be the best way to go for the future, since it is a
> more general approach than mnemosyne was.

[Thomas Lindgren:]
> A related question, to whoever it may concern: why were QLCs
> introduced? Why not fix mnemosyne instead? To the casual user (that
> is, me), they seem quite similar.

Given the poor performance experienced by customers (*) as well as the
non-existing knowledge of Mnemosyne at OTP, product management came to
the decision to either improve Mnemosyne significantly or replace it.
It turned out to be easier to replace it: Mnemosyne creates lots of
processes sending partial answers between them, which is quite slow
and hard to change. The replacement (QLC) focuses on efficient access
of RAM tables (ETS).

Big databases is probably beyond the scope of QLC. The same goes for
advanced query optimization. A more down-to-earth ambition is to
provide the user with information about tables and let him decide the
order of joins when several tables are involved. There are no plans
right now to implement such functionality. In fact, since the existing
functionality covers what (paying) customers use Mnemosyne for,
further work on QLC has been given low priority. It is uncertain if
the promise about fast join of two tables in R11 can be kept
(currently QLC can do joins, but only very inefficiently using nested
loops). To some extent it depends on the interest in QLC from paying
customers.

Best regards

Hans Bolinder, Erlang/OTP

(*) Some (paying) customers use Mnemosyne, but only for trivial tasks:
their queries typically select data from one single table. Whenever
performance has been important, it has so far been possible to rewrite
the queries using match specifications and select statements.