early warning - new rdbms

Fri Mar 3 15:15:48 CET 2006

I can upload a new snapshot.
I'll put it into jungerl pretty soon.

32-bit Erlang can only address 4 GB.
You can either run 64-bit Erlang in order
to address more, or run multiple Erlang nodes.
Very large tables can be managed with 
fragmentation.

For 64-bit erlang, it's important to remember
that the word size is doubled, which means that
many data structures will occupy roughly twice
as much memory as in the 32-bit version.
Binaries work well, whereas strings are a 
veritable disaster (128 bits per character.)

The recent addition of input/output filters
to rdbms might help those who store large 
strings in mnesia and want to use 64-bit erlang
(how large a community is that, I wonder?)

/Ulf W

> -----Original Message-----
> From: Eranga Udesh [mailto:casper2000a@REDACTED] 
> Sent: den 3 mars 2006 14:50
> To: erlang-questions@REDACTED
> Cc: Ulf Wiger (AL/EAB)
> Subject: RE: early warning - new rdbms
> 
> Hi,
> 
> No New RDBMS yet?
> 
> Also regarding the per process memory issue in Linux/Unix 
> came up in the mailing list lately, how can we deal with a 
> large Erlang Mnesia DB, which may have to hold data, say over 
> 5-6 GB of size (5 million records)? Since mnesia disk_copies 
> DB keeps all the data in memory, it's going to be a problem, isn't it?
> 
> Regards,
> - Eranga
> 
> 
> 
> -----Original Message-----
> From: Ulf Wiger (AL/EAB) [mailto:ulf.wiger@REDACTED]
> Sent: Friday, February 24, 2006 3:03 PM
> To: Eranga Udesh
> Subject: RE: early warning - new rdbms
> 
> 
> I'm hoping to be able to upload a snapshot 
> sometime this week. It will not be fully 
> functional, but the integrity checking 
> should work, at least.
> 
> Regards,
> Ulf W 
> 
> > -----Original Message-----
> > From: Eranga Udesh [mailto:casper2000a@REDACTED] 
> > Sent: den 24 februari 2006 04:35
> > To: Ulf Wiger (AL/EAB)
> > Subject: RE: early warning - new rdbms
> > 
> > Hi,
> > 
> > Where's the RDBMS system you talk about? Could I get quick 
> > access, since I am designing a system and I prefer to design 
> > it on top of a RDBMS system, instead of a local Mnesia DB.
> > 
> > Pls advice asap.
> > 
> > Thanks,
> > - Eranga
> > 
> > 
> > 
> > -----Original Message-----
> > From: owner-erlang-questions@REDACTED
> > [mailto:owner-erlang-questions@REDACTED] On Behalf Of Ulf 
> > Wiger (AL/EAB)
> > Sent: Thursday, February 09, 2006 9:15 PM
> > To: Chaitanya Chalasani
> > Cc: erlang-questions@REDACTED
> > Subject: RE: early warning - new rdbms
> > 
> > 
> > Ok, but I've been sidetracked for a few days.
> > I'm still doing some cleanups. I will let you all know as 
> > soon as I have something.
> > 
> > (Again, I was unprepared for so many takers.
> > I had expected to have to announce it a few times before 
> > anyone took the bait.  ;)
> > 
> > Regards,
> > Ulf W 
> > 
> > > -----Original Message-----
> > > From: Chaitanya Chalasani [mailto:chaitanya.chalasani@REDACTED]
> > > Sent: den 9 februari 2006 13:44
> > > To: Ulf Wiger (AL/EAB)
> > > Cc: erlang-questions@REDACTED
> > > Subject: Re: early warning - new rdbms
> > > 
> > > I would like to beta-test as well.
> > > 
> > > On Wednesday 25 January 2006 14:16, Ulf Wiger (AL/EAB) wrote:
> > > > I thought I'd let the cat out of the bag a little...
> > > >
> > > > If anyone wants to beta-test or help out with some of the more 
> > > > advanced problems, let me know.
> > > >
> > > >
> > > > I've come pretty far along with a new version of rdbms. It
> > > has several
> > > > nice features, and I think it's about to make the 
> transition from 
> > > > 'somewhat interesting' to 'actually useful':
> > > >
> > > > - JIT compilation of verification code. The overhead
> > > >   for type and bounds checking is now only a few (~5)
> > > >   microseconds per write operation.
> > > > - The parameterized indexes that I hacked into mnesia
> > > >   before are now part of rdbms. This include ordered
> > > >   indexes and fragmented indexes (i.e. hashed on
> > > >   index value - so they should scale well.)
> > > > - Rdbms will handle fragmented tables transparently
> > > >   (And actually handles plain tables with less overhead
> > > >   than mnesia_frag does.) The overhead for using the
> > > >   rdbms access module (compared to no access module)
> > > >   on a plain transaction is in the order of 20
> > > >   microseconds on my 1 GHz SunBLADE.
> > > > - Rdbms hooks into the transaction handling in such a
> > > >   way that it can automatically rebuild the verification
> > > >   code as soon as a schema transaction commits.
> > > > - A readable file format for schema definitions, trying
> > > >   to establish a structured way to create large mnesia
> > > >   databases. I've also added a 'group' concept to be
> > > >   able to group tables into corresponding subsystems,
> > > >   since I thought this might be helpful in large
> > > >   systems.
> > > >
> > > >
> > > > I'm planning to release rdbms with OTP R11, since it 
> > requires some 
> > > > changes to mnesia that (hopefully) will make it into 
> R11. R11 is 
> > > > planned for May.
> > > >
> > > > Some of the (fairly minor) changes to mnesia so far:
> > > >
> > > > - The access module can hook into the transaction
> > > >   flow by providing callbacks for begin_activity()
> > > >   and end_activity(). Rdbms uses this for proper
> > > >   handling of abort and commit triggers as well as
> > > >   loop detection in referential integrity checks.
> > > >   It also allows rdbms to detect schema changes.
> > > > - An 'on-load' hook allows rdbms to build indexes
> > > >   the first time a table is loaded.
> > > > - A low-level access API for foreign tables. My
> > > >   first foreign table attempt was a 'disk_log'.
> > > >   It makes it possible to properly log events
> > > >   inside a transaction context. You also get
> > > >   replicated logs almost for free, as well as
> > > >   (if you want to) fragmented logs. (:
> > > >   My next attempt at a foreign table is a read-
> > > >   only file system (doesn't have to be read-only,
> > > >   but I thought I'd start with that.) Thus
> > > >   the experiments with converting regexps to
> > > >   the select pattern syntax.
> > > >
> > > > One interesting experiment might be to define an ISAM table
> > > type for
> > > > really large ordered sets on disk. Combining it with 
> > rdbms, you can
> > > > type- specify the data types and then convert to whatever 
> > format is 
> > > > expected by the ISAM files.
> > > >
> > > > Some questions to those who might be interested:
> > > >
> > > > - I'd like to break compatibility with the old
> > > >   rdbms in some respects. Is this a problem for
> > > >   anyone? (speak now or forever hold your peace)
> > > > - Do you have any suggestions or feature requests?
> > > > - Do you want to help out?
> > > >
> > > > Regards,
> > > > Uffe
> > > 
> > > --
> > > Chaitanya Chalasani
> > > 
> > 
> > 
> > 
> 
> 
> 