New trading systems platform

[Ulf Wiger (AL/EAB)]

Vlad Balin wrote:
>
> Well, i would say mnesia should have BTree-based storage 
> option then :),

Disk-based BTree-based storage, you mean? Otherwise, 
Mnesia already has them, in ordered_set tables. Ram_copies
and disc_copies can be ordered_set, but not disc_only_copies.

To make efficient ordered_set disc_only_copies for Mnesia,
I think some restrictions on at least the key are in 
order. It's very difficult to implement BTrees efficiently
if you can't make any assumptions about type and size 
of the keys, or even that they are of the _same_ type.

While perhaps "un-erlangish", it's usually not a problem
to to this in database applications. I think this is one
of the best ways to boost disk performance in Mnesia.

> and it should support clustered index.

Off-the-shelf mnesia doesn't do this, but I have a 
hacked version that does. Even my version doesn't 
support globally ordered clustered indexes, though.
I haven't figured out a good algorithm for that
(the trick being knowing where to put the partition
splits -- again, without being able to make any 
assumptions about the types of data used as index 
values.)

> It should be able to work with huge databases without
> tricks like partitioning single table into several
> pieces.

Why? If your alternative is to circumvent mnesia, 
inventing various tricks to do this, or lots of extra
programming to interface with some SQL database -- why
impose this purity restriction on mnesia?

Partitioning tables in Mnesia is not that difficult.
I don't know enough about trading systems to know whether
it creates especially severe problems in this case,
but for the most part, the user of a mnesia table is 
not aware of whether or not it is partitioned.

> 
> Easiest way to achieve that would be to use Berkley DB
> http://www.sleepycat.com/ as a storage ;)
> 
> > A final thought. Performance in disk-based accesses can
> > be improved significantly if one can nail down the
> > data format. This is something I think should be an option
> > in mnesia.
> Here's a bit different option. User might want to store data by
> chunks in custom [compressed] format. Each chunk is being
> compressed separately, and contains range of values. It
> would be helpful to do something to allow this stuff
> with chunks to be customized and accessed transparently, as 
> if it would be plain table with no chunks.

This is quite possible to do today, based on what's in mnesia,
or ets, dets or disk_log, for that matter. Storing binary 
payload in either of these formats is a piece of cake.
In order to offer a nice interface to the application
programmer, some (in some cases quite a bit of) coding is 
probably needed.

/Uffe