Loading Mnesia dbase

[Ulf Wiger]

First of all, try enabling the thread pool.
I would pick a nice round number like 255, i.e.
erl +A 255.

Also, is your erlang node running out of physical
RAM? It seems like it wouldn't, with 2 GB RAM, but
I guess if you keep items both in process space and
in mnesia, there would be a chance... Anyway, you
can easily track memory use using 'top' or 'vmstat'.

/Uffe

Den 2005-11-18 20:13:39 skrev <tty@REDACTED>:

> Hello,
>
> I have total of 14 million entries in several files to load into a  
> Mnesia dbase. I used one process per file and found that the first 10  
> million entries took around 15 mins to load into Mnesia (ram_copy).  
> However after this initial 10 million entries things started to crawl. A  
> 'ps' shows beam mainly blocking in I/O with around 3% CPU usage.
>
> I then restarted the test with a table of 15 fragments (ram_copy)  
> thinking I hit some Mnesia limit. This dropped my initial 10 million  
> entries to 8.5 mins. However its now back down to a crawl. The last  
> 72000 entries took over 40 mins.
>
> I'm running on SuSe 10 (64bits), dual AMD Opteron 246 with 2GB RAM. I  
> have 5 processes with 3 million entries each which is around 50MB of  
> disc space per file. Each entry is a list of 6 integers. Each record in  
> the dbase has 7 integers. All processes and Mnesia are running of one  
> instance of the VM. Also using dirty_write's.
>
> Does anyone have any suggestions on speeding this ?
>
> Thanks
>
> t



-- 
Ulf Wiger