Batch-loading Mnesia

Joel Reymont <>
Wed Jul 13 11:22:25 CEST 2005


Folks,

It's taking me way over an hour (maybe two hours or three hours) to  
load 119,275 records into Mnesia. Parsing the same file without  
adding record takes about 6 seconds.

I'm doing everything on the same node, adding record sequentially. Is  
there a way to speed things up?

The table has the following format

-record(tick, {
       symbol,
       datetime,
       timestamp,
       price,
       size
      }).

Symbol is a string (list) of 4 characters, datetime is a bignum  
formed from a date and time 20040901,09:39:38 by stripping the comma  
and the colons. Timestamp is now(), price and size are float and  
integer respectively.

Table is created like this:

mnesia:create_table(tick,
                  [
                   {disc_copies, Nodes},
                   {index, [datetime]},
                   {type, bag},
                   {attributes, record_info(fields, tick)}])

The function that adds a record to the database looks like this:

add_tick(Symbol, Date, Time, Price, Size) ->
     Tick = #tick {
       symbol = Symbol,
       datetime = Date * 1000000 + Time,
       price = Price,
       size = Size,
       timestamp = now()
      },
     %%F = fun() -> mnesia:write(Tick) end,
     %%mnesia:transaction(F).
     mnesia:dirty_write(Tick).

--
http://wagerlabs.com/uptick






More information about the erlang-questions mailing list