Loading Mnesia dbase

[tty@REDACTED]

Thanks Ulf,

I'll give the thread pool a try. I'm maxing out at 1.8 GB RAM so everthing is still nicely in RAM space.

Thanks.

t

-------- Original Message --------
From: "Ulf Wiger" <ulf@REDACTED>
To: tty@REDACTED, erlang-questions@REDACTED
Subject: Re: Loading Mnesia dbase
Date: Fri, 18 Nov 2005 21:53:37 +0100

> 
> First of all, try enabling the thread pool.
> I would pick a nice round number like 255, i.e.
> erl +A 255.
> 
> Also, is your erlang node running out of physical
> RAM? It seems like it wouldn't, with 2 GB RAM, but
> I guess if you keep items both in process space and
> in mnesia, there would be a chance... Anyway, you
> can easily track memory use using 'top' or 'vmstat'.
> 
> /Uffe
> 
> Den 2005-11-18 20:13:39 skrev <tty@REDACTED>:
> 
> > Hello,
> >
> > I have total of 14 million entries in several files to load into a  
> > Mnesia dbase. I used one process per file and found that the first 10  
> > million entries took around 15 mins to load into Mnesia (ram_copy).  
> > However after this initial 10 million entries things started to crawl. A  
> > 'ps' shows beam mainly blocking in I/O with around 3% CPU usage.
> >
> > I then restarted the test with a table of 15 fragments (ram_copy)  
> > thinking I hit some Mnesia limit. This dropped my initial 10 million  
> > entries to 8.5 mins. However its now back down to a crawl. The last  
> > 72000 entries took over 40 mins.
> >
> > I'm running on SuSe 10 (64bits), dual AMD Opteron 246 with 2GB RAM. I  
> > have 5 processes with 3 million entries each which is around 50MB of  
> > disc space per file. Each entry is a list of 6 integers. Each record in  
> > the dbase has 7 integers. All processes and Mnesia are running of one  
> > instance of the VM. Also using dirty_write's.
> >
> > Does anyone have any suggestions on speeding this ?
> >
> > Thanks
> >
> > t
> 
> 
> 
> -- 
> Ulf Wiger