dets improvements?

[Hans.Bolinder@REDACTED]

[Yariv Sadan:]
> I hope I'm not bringing it up too many times but I figured I'll give
> one last shot trying to get an answer... Is the OTP team planning on
> making the dets improvements discussed in this thread?

There are currently no plans to improve Dets.

> Maybe it's possible to develop for Mnesia an alternative disc only 
> storage engine designed for storing large blobs. 

Dets stores objects on the external term format, which means that
binary_to_term() (term_to_binary()) is called whenever objects are
retrieved (stored). These functions block the emulator (or just the
scheduler running the Dets process, if SMP), which spoils the soft
real-time properties for huge terms. Films etc are better stored
outside of Dets.

[Klacke:]
> Two major problems with dets as of today (and of yeasterday)
> 
> 1. 64 bit indexing - really easy to fix
> 2. repair time - this is a bit harder to fix, ...

As regards the second point: 

The repair time was significantly reduced in Erlang/OTP R8B. When
repairing, data is written and read serially; no random access is
involved. (By the way, the same goes for copying (open) Dets files
between nodes, something Mnesia does every now and then.) A full Dets
table (16 millions small objects) should not take more than half an
hour to repair, at the worst. This is of course a very long time... 
I don't think it's easy to further reduce repair times.

The major problem with Dets as I see it is that the memory allocation
scheme (a buddy system) is kept in RAM. For a fragmented table with
millions of objects, the RAM data can amount to several megabytes.
When closing or syncing a table, this (possibly huge) data structure
has to be written to disc.

Best regards,

Hans Bolinder, Erlang/OTP