Batch-loading Mnesia

[Joel Reymont]

Folks,

It's taking me way over an hour (maybe two hours or three hours) to  
load 119,275 records into Mnesia. Parsing the same file without  
adding record takes about 6 seconds.

I'm doing everything on the same node, adding record sequentially. Is  
there a way to speed things up?

The table has the following format

-record(tick, {
       symbol,
       datetime,
       timestamp,
       price,
       size
      }).

Symbol is a string (list) of 4 characters, datetime is a bignum  
formed from a date and time 20040901,09:39:38 by stripping the comma  
and the colons. Timestamp is now(), price and size are float and  
integer respectively.

Table is created like this:

mnesia:create_table(tick,
                  [
                   {disc_copies, Nodes},
                   {index, [datetime]},
                   {type, bag},
                   {attributes, record_info(fields, tick)}])

The function that adds a record to the database looks like this:

add_tick(Symbol, Date, Time, Price, Size) ->
     Tick = #tick {
       symbol = Symbol,
       datetime = Date * 1000000 + Time,
       price = Price,
       size = Size,
       timestamp = now()
      },
     %%F = fun() -> mnesia:write(Tick) end,
     %%mnesia:transaction(F).
     mnesia:dirty_write(Tick).

--
http://wagerlabs.com/uptick