call for testers: compound indexes in mnesia
Ulf Wiger (AL/EAB)
Mon May 16 23:27:06 CEST 2005
I mentioned earlier about my hack to add more flexible
indexes to mnesia
I've now come up with a solution that I'm fairly happy
with myself, and even though there are some finishing
touches left, I'd like to solicit some beta testers.
This is what works:
- Indexes are now first-class tables, although flagged
so that they can't be accidentally written to.
- You can have multiple, named, indexes on the same
attribute, or indexes on the whole object. One
unnamed index is allowed per attribute (compatible
with the old indexes).
- Indexes can be ordered or unordered. Ordered disc_only
indexes are obviously not allowed.
- For each index, a callback function is specified.
This function is called with either the attribute
value, or the whole object. It returns a list of
index values, and must be entirely side-effect free.
This means you can have case-insensitive word or
word stem indexes, or index on combinations of
different attributes ("compound indexes" --
- All updates to indexes are visible during the
transaction in which they were written, but not
outside of it, until committed. This is how it
works today as well, but the current solution
didn't work for the more flexible indexes.
- Index tables inherit attributes from the main table
by default: a disc_copy table gets disc_copy indexes
with the same replication pattern, etc.
A fragmented table gets fragmented indexes, with the
same number of fragments (but hashed on index value)
An indexed read on a fragmented table will try to
pick the right index fragment, fetch the keys from
the index, and then perform a multicall to the nodes
that have fragments containing keys of interest.
(Some optimizations could be done, but this should be
a *huge* improvement compared to the existing solution)
- It's possible to specify individual table properties
for each index table. For example, you can specify a
centralized index table for a fragmented main table.
This could work if the index is much smaller than the
main table, in terms of memory useage.
- table loads in various configurations
- transaction writes and index_reads
- dirty writes and index_reads
- index_match_object, index_first, index_last, index_fold
- reads and writes on fragmented table with a fragmented
- reads and writes on a table where table and index reside
on different nodes.
Things that remain:
- Some more sanity checks when creating tables and indexes
- Conversion of existing databases to the new format
and support for things like transform_table
- The remaining functions in mnesia_frag
- Looking at the snmp connection, which should become an
ordered index on the whole object. This should allow
for more flexible snmp mappings and also allow snmp
mappings toward fragmented tables (not possible today)
- Writing a library of useful index callbacks.
- (It's basically possible now to have indexed indexes,
but why would one want to? Forbid, or detect loops?)
- Cleaning up the code. I think this change actually
simplifies the handling of indexes in mnesia.
- Double-check -- triple-check -- that table loading
is done right. Mnesia's table loading code is a
scary place for a weekend hacker.
- Run the whole mnesia test suite
Let me know if you want to help shake out the bugs.
More information about the erlang-questions