Suggestion: New table iterators

[Hakan Mattsson]

On Thu, 19 Oct 2000, Sean Hinde wrote:

Sean> I'd like to add my support for iterators over ets/dets/mnesia.

Yes, at least they looks cute. But in order to enable a real efficient
implementation of them, the not yet released select functionality in
ets and dets is needed. Then it would be possible to efficiently
perform parts of the filtering, locally, near the data source,
avoiding lots of copying. With interruptable semantics like
ets:db_match_object/3, extended with the possibility to access the
interim result in each iteration chunk it would really open up for
blinding fast Mnesia iterators.

This could of course be combined with cute list-like iterators, where
a fun only is applied on those records that fulfills the first select
constraints. 

In order to hide the efficient low level select constraints it would 
be possible to let the Mnemosyne compiler generate them from its
nice list comprehension syntax.

Sean> At the moment writing a safe iterator over an mnesia table is not obvious if
Sean> you want to also have concurrent updates. Apart from anything else there
Sean> appears to be no guarantee that a table will not be re-hashed during the
Sean> iteration - even in a transaction.
Sean> 
Sean> If you want to delete stuff during the iteration it is even less obvious how
Sean> to do this safely and simply. Building a list of stuff to be deleted later
Sean> can chew up masses of memory..

The trouble with dynamic rehashing of Mnesia tables is only a problem
if you perform dirty updates during the iteration.

You may safely iterate over a table in a transaction iff no other
process concurrently performs dirty updates of the same table.

Inside the transaction you must first grab a lock on the entire table,
before you start iterating over the table with mnesia:dirty_first and
mnesia:dirty_next. The key lookups may safely be performed with
mnesia:read or mnesia:dirty_read. If the mnesia:read does not return
any records you may happily continue with the next key, since it means
that you have deleted the record inside the enclosing transaction.  Do
NOT use the mnesia:dirty* functions to perform updates during the
iteration. Use the transaction protected equivalents functions
instead.

/Håkan