New trading systems platform

[Ulf Wiger (AL/EAB)]

Vlad Balin wrote:
>
> In case of ATS we need disk tick storage for the backtesting 
> purpose only. It implies very simple access scenario (rare 
> look-ups, sequential [forward in time] read of large data 
> volumes, "joins" of several tables by time field). It make 
> sense to develop file-based storage for tick storage (one
> file per instrument, data stored sequentially by chunks of 
> compressed ticks), and avoid mnesia.

Perhaps mnesia should be avoided... I'd still like to 
explore ways to keep it in the game though.  (:

The best way to dump large sets of tuples to disk in mnesia,
if sequential storage is needed, is disk_log. disk_log:chunk()
allows you to fetch one object at a time. It's hardly going 
to compete performance-wise with a custom solution written 
in C/C++.

...or could it? We've seen several times that total system
performance in Erlang often compares very well with that
of similar systems written in C. This gets more obvious the
more complex the application is, and perhaps especially
the more concurrency that needs to be handled.

One potential problem here is if there is some frequently
executed "tight loop" that is difficult to make fast in 
erlang.

Storing data in RAM, in erlang format, helps reduce the 
semantic gap. Mnesia and ets have some nice fold functions.
Does the data always get accessed sequentially, or are 
queries also needed?

How about disc_copies (which are stored using disk_log
on disk) that are cleared out of RAM when they get old
enough? When accessed, they'd get loaded again and this
fully transparent to the programmer.

I foresee some mind-expanding modifications of the core
mnesia table loading code -- good fun!

A final thought. Performance in disk-based accesses can
be improved significantly if one can nail down the
data format. This is something I think should be an option
in mnesia.

/Uffe