[erlang-questions] Re: Constantly archiving old mnesia records

[Bernard Duggan]

Actually, this is pretty similar to the problem we're going to be
dealing with soon (except we shouldn't ever have duplicates).  We don't
need to archive the data at the moment, so I'm just running a sweeper
process to clean up stuff older than a certain age, but that won't be
the case in future versions.

I have similar fears to you regarding holding table locks for a long
period - not sure how it's going to pan out in practise under production
loads - it's been not too bad at all in testing though.

Currently the tack we're thinking of taking is to shift old stuff from
erlang to either an SQL or perhaps couchDB system.  This is possible in
our system because the "old" and "current" data are likely to be
accessed for totally separate and different purposes, so there's no need
to provide some common interface or replicate anything.  Always nice to
hear other people's ideas on the matter though.

Cheers,

Bernard

adam wrote:
> The solution I came up with:
>
> Every week, create a new table. Use this week's table and last week's
> table to check for duplicates. Each week, remove the table from the
> week before last.
>
> I created a table_manager server with a timer:interval to manage the
> table rotation.
>
> Hope this helps someone else (though I'm guessing it probably won't)!
>
> Adam
>
>
> On Nov 16, 7:34 pm, adam <adus...@REDACTED> wrote:
>   
>> I have an application in which I have several million new records
>> coming in each day, and I need to make sure that no duplicate ids get
>> past a certain point in the system. To do this, I'm using a
>> distributed mnesia database.
>>
>> Duplicates can only occur within a few days of each other, so I want
>> to ship all old ids off to an archive every week. I could do this by
>> simply selecting old records, iterating through, saving them somewhere
>> else, and deleting them, but I'm afraid of this locking the entire
>> table, and I'm also afraid that this will cause the db file to get
>> pretty bloated over time.
>>
>> The other way to do it is to make a module implementing the
>> mnesia_frag_hash callback behavior which would create a fragment
>> containing old records when I call mnesia:change_table_frag with
>> add_frag. Then perhaps I could just move the data using the filesystem
>> (?).
>>
>> Any suggestions for doing this?
>>
>> Thanks in advance!
>> Adam
>>
>> ________________________________________________________________
>> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
>> erlang-questions (at) erlang.org
>>     
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>