early warning - new rdbms
Ulf Wiger (AL/EAB)
Wed Jan 25 09:46:20 CET 2006
I thought I'd let the cat out of the bag a little...
If anyone wants to beta-test or help out with some
of the more advanced problems, let me know.
I've come pretty far along with a new version of
rdbms. It has several nice features, and I think
it's about to make the transition from 'somewhat
interesting' to 'actually useful':
- JIT compilation of verification code. The overhead
for type and bounds checking is now only a few (~5)
microseconds per write operation.
- The parameterized indexes that I hacked into mnesia
before are now part of rdbms. This include ordered
indexes and fragmented indexes (i.e. hashed on
index value - so they should scale well.)
- Rdbms will handle fragmented tables transparently
(And actually handles plain tables with less overhead
than mnesia_frag does.) The overhead for using the
rdbms access module (compared to no access module)
on a plain transaction is in the order of 20
microseconds on my 1 GHz SunBLADE.
- Rdbms hooks into the transaction handling in such a
way that it can automatically rebuild the verification
code as soon as a schema transaction commits.
- A readable file format for schema definitions, trying
to establish a structured way to create large mnesia
databases. I've also added a 'group' concept to be
able to group tables into corresponding subsystems,
since I thought this might be helpful in large
I'm planning to release rdbms with OTP R11, since it
requires some changes to mnesia that (hopefully) will
make it into R11. R11 is planned for May.
Some of the (fairly minor) changes to mnesia so far:
- The access module can hook into the transaction
flow by providing callbacks for begin_activity()
and end_activity(). Rdbms uses this for proper
handling of abort and commit triggers as well as
loop detection in referential integrity checks.
It also allows rdbms to detect schema changes.
- An 'on-load' hook allows rdbms to build indexes
the first time a table is loaded.
- A low-level access API for foreign tables. My
first foreign table attempt was a 'disk_log'.
It makes it possible to properly log events
inside a transaction context. You also get
replicated logs almost for free, as well as
(if you want to) fragmented logs. (:
My next attempt at a foreign table is a read-
only file system (doesn't have to be read-only,
but I thought I'd start with that.) Thus
the experiments with converting regexps to
the select pattern syntax.
One interesting experiment might be to define
an ISAM table type for really large ordered sets
on disk. Combining it with rdbms, you can type-
specify the data types and then convert to
whatever format is expected by the ISAM files.
Some questions to those who might be interested:
- I'd like to break compatibility with the old
rdbms in some respects. Is this a problem for
anyone? (speak now or forever hold your peace)
- Do you have any suggestions or feature requests?
- Do you want to help out?
More information about the erlang-questions