Loading Mnesia dbase

[Sean Hinde]

>
> I'll give the thread pool a try. I'm maxing out at 1.8 GB RAM so  
> everthing is still nicely in RAM space.

That doesn't leave much room for everything else running on the  
machine. These symptoms do tend to indicate that you are going into  
swap.

Sean


>
> -------- Original Message --------
> From: "Ulf Wiger" <ulf@REDACTED>
> To: tty@REDACTED, erlang-questions@REDACTED
> Subject: Re: Loading Mnesia dbase
> Date: Fri, 18 Nov 2005 21:53:37 +0100
>
>>
>> First of all, try enabling the thread pool.
>> I would pick a nice round number like 255, i.e.
>> erl +A 255.
>>
>> Also, is your erlang node running out of physical
>> RAM? It seems like it wouldn't, with 2 GB RAM, but
>> I guess if you keep items both in process space and
>> in mnesia, there would be a chance... Anyway, you
>> can easily track memory use using 'top' or 'vmstat'.
>>
>> /Uffe
>>
>> Den 2005-11-18 20:13:39 skrev <tty@REDACTED>:
>>
>>> Hello,
>>>
>>> I have total of 14 million entries in several files to load into a
>>> Mnesia dbase. I used one process per file and found that the  
>>> first 10
>>> million entries took around 15 mins to load into Mnesia (ram_copy).
>>> However after this initial 10 million entries things started to  
>>> crawl. A
>>> 'ps' shows beam mainly blocking in I/O with around 3% CPU usage.
>>>
>>> I then restarted the test with a table of 15 fragments (ram_copy)
>>> thinking I hit some Mnesia limit. This dropped my initial 10 million
>>> entries to 8.5 mins. However its now back down to a crawl. The last
>>> 72000 entries took over 40 mins.
>>>
>>> I'm running on SuSe 10 (64bits), dual AMD Opteron 246 with 2GB  
>>> RAM. I
>>> have 5 processes with 3 million entries each which is around 50MB of
>>> disc space per file. Each entry is a list of 6 integers. Each  
>>> record in
>>> the dbase has 7 integers. All processes and Mnesia are running of  
>>> one
>>> instance of the VM. Also using dirty_write's.
>>>
>>> Does anyone have any suggestions on speeding this ?
>>>
>>> Thanks
>>>
>>> t
>>
>>
>>
>> -- 
>> Ulf Wiger