time for deleting records from mnesia

[Tony Zheng]

Hi, Ulf

I used mnesia:activity(async_dirty, F) instead of mnesia:transaction(F),
it was really faster than before. It just spent 1 minutes when deleting
100000 records.  
 
    F = fun() ->		
	MatchHead = #t_transaction{subrecord_id='$1', transaction_time='$2',
_='_'},
	GuardDel = {'==', {element, 1, '$2'}, {date()}}, 
	Result = '$_',
	ListsDel = mnesia:select(t_transaction,[{MatchHead, [GuardDel],
[Result]}]),
	lists:foreach(fun(List) -> mnesia:delete_object(List) end, ListsDel)
    end,
    mnesia:activity(async_dirty, F)

When it works, some of error messages displayed:

    =ERROR REPORT==== 9-May-2006::11:05:41 ===
    Mnesia(one@REDACTED): ** WARNING ** Mnesia is overloaded: {dump_log,
                                                            
write_threshold}

    =ERROR REPORT==== 9-May-2006::11:05:42 ===
    Mnesia(one@REDACTED): ** WARNING ** Mnesia is overloaded: {dump_log,
                                                            
write_threshold}

    =ERROR REPORT==== 9-May-2006::11:05:42 ===
    Mnesia(one@REDACTED): ** WARNING ** Mnesia is overloaded: {dump_log,
                                                            
write_threshold}

I checked the mnesia table after deleting, it deleted all selected
records and seemed to work well. I am not sure if I can ignore these
error messages? Does it mean some failure to manipulate the mnesia
table? Will it effect the data in mnesia table? Thanks.

Best regards

tony      



On Tue, 2006-05-09 at 00:57, Ulf Wiger (AL/EAB) wrote:
> When deleting 100000 records inside one transaction,
> mnesia has to check the transaction store for each
> delete.
> 
> Just to separate what is overhead from transaction-
> protected delete and what is storage overhead,
> try doing the same thing with dirty access.
> 
> Hint: If you use mnesia:activity(Type, F) instead
> of mnesia:transaction(F), you only need to change
> Type from 'transaction' to e.g. 'async_dirty'. The
> funs you've written will work in either case.
> 
> BR,
> Ulf W
> 
> > -----Original Message-----
> > From: owner-erlang-questions@REDACTED 
> > [mailto:owner-erlang-questions@REDACTED] On Behalf Of Tony Zheng
> > Sent: den 9 maj 2006 02:52
> > To: erlang-questions@REDACTED
> > Subject: time for deleting records from mnesia
> > 
> > Hi
> > 
> > I want to delete some selected records from my mnesia 
> > database table(t_test). I tried to use 
> > mnesia:delete_object(Record) and mnesia:delete({Tab, Key}), 
> > but both of them needed about 20 minutes when I deleted 
> > 100000 records. I want to know if there are some other method 
> > to do it faster? Thanks.
> > 
> > 1. use mnesia:delete_object(Record) to delete 100000 records, it spent
> > 22 minutes.
> > 
> >    F = fun() ->		
> > 	MatchHead = #t_test{transaction_time='$1', _='_'},
> > 	GuardDel = {'==', {element, 1, '$1'}, {date()}}, 
> > 	Result = '$_',
> > 	ListsDel = mnesia:select(t_test,[{MatchHead, 
> > [GuardDel], [Result]}]),
> > 	lists:foreach(fun(List) -> mnesia:delete_object(List) 
> > end, ListsDel)
> >    end,
> >    mnesia:transaction(F).
> > 
> > 2. use mnesia:delete({Tab, Key}) to delete 100000 records, it 
> > spent 23 minutes.
> >  
> >    F = fun() ->		
> > 	MatchHead = #t_test{transaction_time='$1', _='_'},
> > 	GuardDel = {'==', {element, 1, '$1'}, {date()}}, 
> > 	Result = '$_',
> > 	ListsDel = mnesia:select(t_test,[{MatchHead, 
> > [GuardDel], [Result]}]),
> > 	lists:foreach(fun(List) -> mnesia:delete({t_test, 
> > element(2, List)}) end, ListsDel)
> >    end,
> >    mnesia:transaction(F).
> > 
> > 
> > tony
> > 
> >