Processing Large Mnesia dbase

[tty@REDACTED]

Hello,

mnesia:foldl did the trick. On my 9 million records dbase it went from 64 mins on 3 nodes to 5.3 mins on 1 node. On my 14 million records it went from 9 hours to 15 mins on 1 node.

Thinking about Mnesia as a traditional database and using read/dirty_read was my downfall. 

Thanks to all who answered my questions.

t

-------- Original Message --------
From: "Ulf Wiger (AL/EAB)" <ulf.wiger@REDACTED>
To: <tty@REDACTED>
Cc: <erlang-questions@REDACTED>
Subject: RE: Processing Large Mnesia dbase
Date: Tue, 29 Nov 2005 18:00:39 +0100

> 
> You need to place the call to mnesia:foldl() from
> within a fun passed to either mnesia:activity() or
> mnesia:transaction(). If you use 
> mnesia:activity(Type, Fun), then Type can be either
> of the valid types (ets, async_dirty, ..., sync_transaction.)
> 
> Example:
> 
> mnesia:activity(
>    transaction,
>    fun() ->
>            mnesia:foldl(fun(X,Acc) -> [X|Acc] end, [], Tab)
>    end).     
>  
> 
> /Uffe
> 
> > -----Original Message-----
> > From: owner-erlang-questions@REDACTED 
> > [mailto:owner-erlang-questions@REDACTED] On Behalf Of 
> > tty@REDACTED
> > Sent: den 29 november 2005 16:56
> > To: Bengt Kleberg (AL/EAB)
> > Cc: erlang-questions@REDACTED
> > Subject: Re: Processing Large Mnesia dbase
> > 
> > Unfortunately I get a 
> > 
> >                ** exited: {aborted,no_transaction} **
> > 
> > when running mnesia:foldl.
> > 
> > I concur it would be great to hear folks with large mnesia 
> > experience on this matter.
> > 
> > Thanks
> > 
> > t
> >