early warning - new rdbms

[Eranga Udesh]

Hi Uffe,

I am seriously interested to beta-test or help with your development. I have
used Mnesia in many development and production environments. Please advice
the next step.

Thanks,
- Eranga



-----Original Message-----
From: owner-erlang-questions@REDACTED
[mailto:owner-erlang-questions@REDACTED] On Behalf Of Ulf Wiger (AL/EAB)
Sent: Wednesday, January 25, 2006 2:46 PM
To: erlang-questions@REDACTED
Subject: early warning - new rdbms


I thought I'd let the cat out of the bag a little...

If anyone wants to beta-test or help out with some
of the more advanced problems, let me know.


I've come pretty far along with a new version of 
rdbms. It has several nice features, and I think
it's about to make the transition from 'somewhat
interesting' to 'actually useful':

- JIT compilation of verification code. The overhead
  for type and bounds checking is now only a few (~5)
  microseconds per write operation.
- The parameterized indexes that I hacked into mnesia
  before are now part of rdbms. This include ordered
  indexes and fragmented indexes (i.e. hashed on 
  index value - so they should scale well.)
- Rdbms will handle fragmented tables transparently
  (And actually handles plain tables with less overhead
  than mnesia_frag does.) The overhead for using the
  rdbms access module (compared to no access module)
  on a plain transaction is in the order of 20 
  microseconds on my 1 GHz SunBLADE.
- Rdbms hooks into the transaction handling in such a
  way that it can automatically rebuild the verification
  code as soon as a schema transaction commits.
- A readable file format for schema definitions, trying
  to establish a structured way to create large mnesia
  databases. I've also added a 'group' concept to be
  able to group tables into corresponding subsystems,
  since I thought this might be helpful in large 
  systems.


I'm planning to release rdbms with OTP R11, since it
requires some changes to mnesia that (hopefully) will
make it into R11. R11 is planned for May.

Some of the (fairly minor) changes to mnesia so far:

- The access module can hook into the transaction
  flow by providing callbacks for begin_activity()
  and end_activity(). Rdbms uses this for proper
  handling of abort and commit triggers as well as
  loop detection in referential integrity checks.
  It also allows rdbms to detect schema changes.
- An 'on-load' hook allows rdbms to build indexes
  the first time a table is loaded.
- A low-level access API for foreign tables. My 
  first foreign table attempt was a 'disk_log'. 
  It makes it possible to properly log events 
  inside a transaction context. You also get 
  replicated logs almost for free, as well as 
  (if you want to) fragmented logs. (:
  My next attempt at a foreign table is a read-
  only file system (doesn't have to be read-only,
  but I thought I'd start with that.) Thus
  the experiments with converting regexps to 
  the select pattern syntax.

One interesting experiment might be to define 
an ISAM table type for really large ordered sets
on disk. Combining it with rdbms, you can type-
specify the data types and then convert to 
whatever format is expected by the ISAM files.

Some questions to those who might be interested:

- I'd like to break compatibility with the old
  rdbms in some respects. Is this a problem for
  anyone? (speak now or forever hold your peace)
- Do you have any suggestions or feature requests?
- Do you want to help out?

Regards,
Uffe