call for testers: compound indexes in mnesia

[Ulf Wiger (AL/EAB)]

I mentioned earlier about my hack to add more flexible 
indexes to mnesia
(http://www.erlang.org/ml-archive/erlang-questions/200502/msg00233.html)

I've now come up with a solution that I'm fairly happy 
with myself, and even though there are some finishing 
touches left, I'd like to solicit some beta testers.

This is what works:

- Indexes are now first-class tables, although flagged 
  so that they can't be accidentally written to.

- You can have multiple, named, indexes on the same 
  attribute, or indexes on the whole object. One 
  unnamed index is allowed per attribute (compatible
  with the old indexes).

- Indexes can be ordered or unordered. Ordered disc_only
  indexes are obviously not allowed.

- For each index, a callback function is specified. 
  This function is called with either the attribute 
  value, or the whole object. It returns a list of 
  index values, and must be entirely side-effect free.
  This means you can have case-insensitive word or
  word stem indexes, or index on combinations of 
  different attributes ("compound indexes" -- 
  extremely useful).

- All updates to indexes are visible during the 
  transaction in which they were written, but not 
  outside of it, until committed. This is how it 
  works today as well, but the current solution
  didn't work for the more flexible indexes.

- Index tables inherit attributes from the main table 
  by default: a disc_copy table gets disc_copy indexes 
  with the same replication pattern, etc.
  A fragmented table gets fragmented indexes, with the
  same number of fragments (but hashed on index value)
  An indexed read on a fragmented table will try to
  pick the right index fragment, fetch the keys from 
  the index, and then perform a multicall to the nodes
  that have fragments containing keys of interest.
  (Some optimizations could be done, but this should be
  a *huge* improvement compared to the existing solution)

- It's possible to specify individual table properties
  for each index table. For example, you can specify a 
  centralized index table for a fragmented main table.
  This could work if the index is much smaller than the
  main table, in terms of memory useage.


I've tested:

- table loads in various configurations
- transaction writes and index_reads
- dirty writes and index_reads
- index_match_object, index_first, index_last, index_fold
- reads and writes on fragmented table with a fragmented
  index
- reads and writes on a table where table and index reside
  on different nodes.

Things that remain:

- Some more sanity checks when creating tables and indexes
- Conversion of existing databases to the new format
  and support for things like transform_table
- The remaining functions in mnesia_frag
- Looking at the snmp connection, which should become an
  ordered index on the whole object. This should allow
  for more flexible snmp mappings and also allow snmp
  mappings toward fragmented tables (not possible today)
- Writing a library of useful index callbacks.
- (It's basically possible now to have indexed indexes, 
  but why would one want to? Forbid, or detect loops?)
- Cleaning up the code. I think this change actually 
  simplifies the handling of indexes in mnesia.
- Double-check -- triple-check -- that table loading 
  is done right. Mnesia's table loading code is a 
  scary place for a weekend hacker.
- Run the whole mnesia test suite

Let me know if you want to help shake out the bugs.

/Uffe