Batch-loading Mnesia

[Sean Hinde]

If you are using the same value of datetime for all rows in your test  
routine then this could lead to that behaviour.

The reason is that all the primary keys must be added to a single  
entry in the index lookup table, which gets slow after a while. Even  
so, that is spectacularly slow compared with my experience of such  
small tables

Sean

On 13 Jul 2005, at 10:22, Joel Reymont wrote:

> Folks,
>
> It's taking me way over an hour (maybe two hours or three hours) to  
> load 119,275 records into Mnesia. Parsing the same file without  
> adding record takes about 6 seconds.
>
> I'm doing everything on the same node, adding record sequentially.  
> Is there a way to speed things up?
>
> The table has the following format
>
> -record(tick, {
>       symbol,
>       datetime,
>       timestamp,
>       price,
>       size
>      }).
>
> Symbol is a string (list) of 4 characters, datetime is a bignum  
> formed from a date and time 20040901,09:39:38 by stripping the  
> comma and the colons. Timestamp is now(), price and size are float  
> and integer respectively.
>
> Table is created like this:
>
> mnesia:create_table(tick,
>                  [
>                   {disc_copies, Nodes},
>                   {index, [datetime]},
>                   {type, bag},
>                   {attributes, record_info(fields, tick)}])
>
> The function that adds a record to the database looks like this:
>
> add_tick(Symbol, Date, Time, Price, Size) ->
>     Tick = #tick {
>       symbol = Symbol,
>       datetime = Date * 1000000 + Time,
>       price = Price,
>       size = Size,
>       timestamp = now()
>      },
>     %%F = fun() -> mnesia:write(Tick) end,
>     %%mnesia:transaction(F).
>     mnesia:dirty_write(Tick).
>
> --
> http://wagerlabs.com/uptick
>
>
>
>