[erlang-questions] how: Purging old records from Mnesia table

[Paul Mineiro]

gleb,

lately i've been maintaining another mnesia table as an ordered_set with
the expiration time as the first element in the tuple of the primary key.
i then periodically iterate over that (with mnesia:first/1 and
mnesia:next/2), terminating the iteration early when possible, which is
most of the time.

of course it means database modifications have to hit two tables, and
unfortunately sometimes introduces the need for transactions where before
dirty operations would do.

so i'm eager to hear a better solution.

-- p

p.z. one thing not clear in your original post ... is this persistent
data?  cuz if so, timers and ets tables seem ill advised.

On Thu, 6 Mar 2008, Gleb Peregud wrote:

> Hello.
>
> I would like to implement some algorithm of purging old data from
> mnesia table. Expiration date is written inside each record (could be
> 0 for permanent data).
>
> I have few ideas how to implement it:
> 0) Every few seconds iterate over whole table and purge old records.
> It is implemented in this way now and it eats about 35% of CPU on
> every purge cycle for about 10 000 entries.
> Will index on expiration field speed up selecting? Is it better to use
> qlc? Will it speed give speed up if i use mnesia:index_match_object()
> ?
>
> 1) Use erlang:send_after() to send messages to purge individual
> records (with special care to timeouts greater then max value of
> unsigned int). Will Erlang runtime handle efficiently potentially up
> to few millions of timers?
>
> 2) Use some sort of queue to manage sorted list of records to be
> purged. Make special process which will receive information about new
> records, insert into this queue and sleep until first record is to be
> purged. Probably gb_trees will handle it. Are there better ADS for
> this task?
>
> I'll be greateful for any advices and responses :)
>
> Best Regards,
> --
> Gleb Peregud
> http://gleber.pl/
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>

Optimism is an essential ingredient of innovation. How else can the
individual favor change over security?

  -- Robert Noyce