[erlang-questions] trading systems

[Serge Aleynikov]

Joel Reymont wrote:
> Kenneth,
> 
> On Oct 1, 2009, at 4:50 PM, Kenneth Lundin wrote:
> 
>> It would be nice if you could explain what the system needs to do and 
>> after that
>> why you think it is a problem to store prices in an ETS-table.
>> [...]
>> Why are you talking about mmapped files? Please explain for a novice
>> in trading systems.
> 
> It's high-frequency trading systems I'm talking about. Where's Serge 
> Aleynikov [1] when you need him?

Still here...

> I'd like to process an incoming price quote, make a decision and submit 
> a trade order with minimal latency. I know that this is normally done 
> using C++ (or OCaml [1]) but I'm wondering how this can be achieved 
> using Erlang.

Don't want to be too repetitive to answers already posted here, but what 
is your definition of "minimal latency"?  If you are talking about 
microseconds, having much logic coded in Erlang will not help you.  At 
the end you will end up with basic configuration/controlling 
functionality in Erlang navigating a low-level component coded in some 
low-level language (unless of course you are willing to invest time in 
writing some domain-specific language for the task, which might be an 
overkill given the scope of your work).

> A simple array of floats or doubles groups data together in memory and 
> is very cache friendly. I figure that an Erlang binary is the closest I 
> can get to an array in Erlang. mmap-ing a file of floats or doubles and 
> wrapping it in a binary should avoid unnecessary copying of data from 
> disk to memory.

Even if that component is fast, you'll still have a bytecode-related 
overhead on the Erlang side reading/writing data to this structure 
through copying.  It is ok (and very convenient) for most applications, 
but for what you are talking about that overhead may be a show stopper.

> I think operating on binaries representing mmap-ed files of price quotes 
> is as close as you can get to low latency in Erlang. I know that 
> premature optimization is the root of all evil but I'd rather have my 
> teeth pulled than optimize Erlang. This is speaking from experience. I 
> figure a much better way is to choose a concept that guarantees fast 
> code from the start.

If you don't want to spend too much time coding C++, since you have 
already done good work of abstracting mnesia backend, why don't you 
create an Erlang driver for this database: www.fastdb.org.  It's based 
on memory mapped files, implemented in C++, is relatively fast compared 
to other in-memory databases (about 250k lookups/s, 50k inserts/s), has 
time-series capabilities, simple C-API.  I implemented FastDB interface 
in several languages, and generally it's good for systems that deal with 
latencies in sub-milliseconds range.  It may suffice for you unless you 
need to stay in the low microseconds realm.

My 2c.

Serge