Loading Mnesia dbase
Fri Nov 18 22:03:27 CET 2005
I'll give the thread pool a try. I'm maxing out at 1.8 GB RAM so everthing is still nicely in RAM space.
-------- Original Message --------
From: "Ulf Wiger" <>
Subject: Re: Loading Mnesia dbase
Date: Fri, 18 Nov 2005 21:53:37 +0100
> First of all, try enabling the thread pool.
> I would pick a nice round number like 255, i.e.
> erl +A 255.
> Also, is your erlang node running out of physical
> RAM? It seems like it wouldn't, with 2 GB RAM, but
> I guess if you keep items both in process space and
> in mnesia, there would be a chance... Anyway, you
> can easily track memory use using 'top' or 'vmstat'.
> Den 2005-11-18 20:13:39 skrev <>:
> > Hello,
> > I have total of 14 million entries in several files to load into a
> > Mnesia dbase. I used one process per file and found that the first 10
> > million entries took around 15 mins to load into Mnesia (ram_copy).
> > However after this initial 10 million entries things started to crawl. A
> > 'ps' shows beam mainly blocking in I/O with around 3% CPU usage.
> > I then restarted the test with a table of 15 fragments (ram_copy)
> > thinking I hit some Mnesia limit. This dropped my initial 10 million
> > entries to 8.5 mins. However its now back down to a crawl. The last
> > 72000 entries took over 40 mins.
> > I'm running on SuSe 10 (64bits), dual AMD Opteron 246 with 2GB RAM. I
> > have 5 processes with 3 million entries each which is around 50MB of
> > disc space per file. Each entry is a list of 6 integers. Each record in
> > the dbase has 7 integers. All processes and Mnesia are running of one
> > instance of the VM. Also using dirty_write's.
> > Does anyone have any suggestions on speeding this ?
> > Thanks
> > t
> Ulf Wiger
More information about the erlang-questions