mnesia: inserting a large number of records

[Sebastian Bello]

Hakan, Vance,

I tried the approach you pointed out, with the following results:
#records inserted    Time (without table lock)    Time (with table lock)
1000                        0:36                                  0:20
2000                        1:55                                  1:10
3000                        4:20                                  2:45
4000                        7:00                                  4:40
5000                       11:00                                 7:15
6000                       15:00                                 6:40 (this
may be a mistake)
8000                       24:00                                 11:30
10000                     35:00                                 19:30

With a large number of records the table lock solution becomes more
attractive.
Despite this, the table lock locks all other interactions on the table,
right? I'll have to evaluate if this is acceptable for my scenario.
Thank you both,
    Sebastian-



----- Original Message -----
From: "Vance Shipley" <vances@REDACTED>
To: "Hakan Mattsson" <hakan@REDACTED>
Cc: "Sebastian Bello" <sebastian@REDACTED>; <erlang-questions@REDACTED>
Sent: Monday, August 01, 2005 3:11 PM
Subject: Re: mnesia: inserting a large number of records


> On Mon, Aug 01, 2005 at 03:50:48PM +0200, Hakan Mattsson wrote:
> }
> }  Vance, did you try table locks? How much did you gain
> }  with your solution?
>
> I did not.  The first part of my solution was to seperate the
> text file parsing out.  That takes twenty seconds itself.
> I should have tried locking the table and then inserting the
> preprocessed data to see if that would have given acceptable
> performance.  Our needs took me in a different direction however.
>
> For our use though we really wanted a facility to change the
> whole table quickly and easily.  Each table is a complete related
> set of data.  We swap out the table for a new version and if there
> is a problem we swap it back.  We can maintain multiple versions
> of the table in backup files and change between them at will.
> Admittedly this is a different problem than Sebastian asked about.
>
>
> -Vance
>
>
>