mnesia, large datasets and key range lookups

[Jouni Rynö]

On Thu, 2004-07-15 at 10:15 +0200, Ulf Wiger wrote:

> 
> One thing you could play around with is to use fragmented tables, with
> one fragment per day, and a customized frag_hash algorithm, selecting
> fragment based on the date. This will allow you to keep a single-table
> view, while being able to narrow your searches based on time.
> 
> Check the reference manual for mnesia_frag_hash to find out how to use
> the callback behaviour for custom fragmentation policies. It's not
> that difficult. There is also some sample code to look at in the
> mnesia source (apologies for not just writing down the solution, but
> I have some children here calling for my attention.)
> 

Took a fast look on the mnesia_frag_hash.erl: that would be simple
Took a fast look on the mnesia_frag.erl: Makes one wonder ...

What I would need to do, is to misuse the fragmentation concept. I could
use some base date (like 20040101), then define the fragment number as
(record_date - base_date). Finally in the year 2014 I would have about
5000 fragments. I could live with that. 
I guess upon adding new data, I should check the date and call the
mnesia:change_table_frag(time_parameters, {add_frag, [db@REDACTED]}) enough
times to have fragments up to present date.

Am I on the right track? The match_spec_to_frag_numbers could become
quite a special case, as it would have to match many of the fancy select
cases I could think about. Writing a real parser for it could be an
overkill :)


regards
	Jouni



-- 

  Jouni Rynö                            mailto://Jouni.Ryno@fmi.fi/
                                        http://www.geo.fmi.fi/~ryno/
  Finnish Meteorological Institute      http://www.fmi.fi/
  Space Research                        http://www.geo.fmi.fi/
  P.O.BOX 503                           Tel      (+358)-9-19294656
  FIN-00101 Helsinki                    FAX      (+358)-9-19294603
  Finland                               priv-GSM (+358)-50-5302903
  
  "It's just zeros and ones, it cannot be hard"