[erlang-questions] clarify: why is Mnesia not fit for large databases

[Ulf Wiger (TN/EAB)]

Dave Rafkind skrev:
> AFAIK mnesia has a hardcoded 2GB limit somewhere due to using 32-bit 
> values in some important place.

Mnesia doesn't, but dets files are currently limited to 2 GB per file,
and mnesia uses dets files for disc_only tables. In a fragmented
disc_only table, that would amount to max 2 GB per fragment (which is
not something that mnesia will check or enforce, so living close to
that limit is inadvisable to say the least).

Dets can also suffer fairly long repair times if it can't be sure that
the file is consistent (was closed in an orderly fashion).

If dets were to be replaced by a version that can handle larger files,
so would Mnesia be able to handle correspondingly larger disk-based
tables.

For really, really big databases, one might desire a more fine-grained
table synchronization method between replicas. Right now, everything
is copied from the most current copy. Another thing to wish for might
be a form of caching for large disk-based tables.

It's reasonable to assume that mnesia can safely handle databases
of a number of gigabytes, perhaps (tounge in cheek) a hundred or
so gigabytes, if one thinks carefully, and the access patterns are
favourable. But I don't know of anyone who actually does that,
and most people who have databases that large tend to not want to
be guinea pigs. (:

Terabyte-size databases ought to be out of the question without
major design effort in mnesia. It would also require a very clever
query planner.

To be clear, mnesia was never designed with this in mind.

BR,
Ulf W