[erlang-questions] Deleting millions of rows in mnesia

[Mazen]

I would agree that this would be a good solution if the operation of emptying the table would happen frequently. However if you only wish to clear it once every month you would probably make your program less efficient since the total amount of time checking for which table to go to everytime you want to read/write would probably take longer then just mnesia:clear_table/1 (or delete/recreate) once a month :)

The real question imho is if the small gain (because I bet its a very small difference) of performing the more efficient operation is a necessary optimization for something that is done once a month.

Anyway... My tests showed:


mazen@REDACTED:~/Desktop/test> erl
Erlang (BEAM) emulator version 5.5.3 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.5.3  (abort with ^G)
1> mnesia:create_schema([]).
ok
2> mnesia:start().
ok
3> l(test).
{module,test}
4> test:create().
{atomic,ok}
5> test:fill(10000000).
ok
6> test:clear_table().
Clear time: 6272344
ok
7> test:fill(10000000).
ok
8> test:delete_table().
Delete/Create time: 6462585
ok
9>


Using this code:


-module(test).
-author('code@REDACTED').
-copyright('Erlang Training & Consulting Ltd.').
-vsn("$Rev$").
-export([create/0,fill/1,clear_table/0,delete_table/0]).

-record(row, {cell1,cell2,cell3,cell4}).

create() ->
    mnesia:create_table(test, [{record_name, row},
                               {attributes, record_info(fields, row)},
                               {type, set}]).

fill(0) -> ok;
fill(N) -> mnesia:dirty_write(test,#row{cell1=now()}), fill(N-1).

clear_table() ->
    T1 = now(),
    mnesia:clear_table(test),
    T2 = now(),
    io:format("Clear time: ~p ~n",[timer:now_diff(T2,T1)]).

delete_table() ->
    T1 = now(),
    mnesia:delete_table(test),
    create(),
    T2 = now(),
    io:format("Delete/Create time: ~p ~n",[timer:now_diff(T2,T1)]).


Enough to convince me anyway that mnesia:clear_table/1 is prefered taking into consideration what Ulf mentioned. What took the longest was the writes (of course).

/Mazen


Guest wrote:

...
What I would do in such a case is two have two identical tables with
different names, say A and B. You have some other config which
indicates which is the "active" table. Any process which needs to
access this table reads the config to figure out whether it should
access A or B. When the time comes for the active table to be purged,
change the config to point to the standby table. Then take all the
time you need to empty the old table.

Chandru
_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://www.erlang.org/mailman/listinfo/erlang-questions
 Post recived from mailinglist

(end of quote)

_________________________________________________________
Post sent from http://www.trapexit.org