safedets

[Hakan Mattsson]

On Wed, 29 Sep 1999, Klacke wrote:

klacke>2. If such a file is not properly closed, it takes ages
klacke>   to repair it.

Yes, repair of dets-files is a pain.

Recently, I introduced a new mechanism (auto_save) to dets which also
aims to avoid the absurd repair times of possibly damaged dets-files.
My approach was to delay the "not properly closed" mark until the
first update actually was performed to the table. Later, when no
updates has been performed for a while, the file was automatically
marked as closed again. This approach makes the probability for repair
lower, but does not eliminate it.

klacke>Nevertheless I attacked the second problem instead
klacke>and came up with the idea to use 2 dets files instead.
klacke>I posted the code to 
klacke>
klacke>http://www.erlang.org/user.html#safedets-1.0
klacke>
klacke>
klacke>Each safedets:open_file/2 makes a directory holding the
klacke>2 detsfiles + a log. safedets files are never repaired.
klacke>If one of the dets files need to be repaired, the other
klacke>doesn't. A regular file copy is performed. This is much much
klacke>faster that a logical dets copy which is performed by the
klacke>repair process.
klacke>
klacke>Another thought for the mnesia guys (Dan+Hakan), maybe
klacke>it's agood idea to apply the same tactics on all
klacke>mnesia files that are disc_only_copies. This way we
klacke>can have really large mnesia databases. 

Another approach is to avoid very large dets-files.
By using the feature of fragmented tables in Mnesia,
a table may be split into lots of fragments where
each fragment is implemented as a normal Mnesia
table. This will keep the dets-files small and therefore
improve the repair time, especially together with the
new auto_save mechanism in dets. 

klacke>Any thoughts ... ??

Your algorithm seems to be very promising, since it totally
eliminates the need for repair.

But, the drawback is performance and disk storage.
  
 - Have you made any measurements?

 - Appending to a log-file is not for free,
   but perhaps neglectible? 

 - What happens with the availability of the dets table
   when the write threshold is reached? Does your algorithm
   allow table access (both reads and updates) during
   the log propagation or will the table access be blocked?

/Håkan