File sharing software

[Yariv Sadan]

On May 3, 2006, at 6:18 PM, Claes Wikstrom wrote:

> Yariv Sadan wrote:
>
>> Maybe it's possible to develop for Mnesia an alternative disc  
>> only  storage engine designed for storing large blobs. It wouldn't  
>> have to  be an Oracle killer. It would just have to handle very  
>> large data  sets, have decent performance, and avoid the  
>> rebuilding cost  associated with broken dets tables. Such a  
>> solution would make Mnesia  more versatile and therefore more  
>> appealing for applications for  which it makes sense to sacrifice  
>> soft real-time performance for  large storage capacity. What do  
>> you think?
>>
>
>
> This is certainly possible, I wrote dets some 15 years ago, and
> it high time to rewrite completely. It's been through several
> enhancements within the OTP group, but the original ideas are still
> there. Linear (P.A Larsson algorithm) hashing for indexing and
> a buddy allocator for space on the file.
>
> Two major problems with dets as of today (and of yeasterday)
>
> 1. 64 bit indexing - really easy to fix
> 2. repair time - this is a bit harder to fix, but one solution
>    could be along the following lines.
>
>   a) keep the index and the freelist in a different file,
>   b) have a file soley for the data. This file could be as easily
>      repaired as a disklog file. One could just chunk through it, one
>      term at a time, this way even large files will be fast to repair.
>   c) when an object is put on the freelist, it needs also to be
>      overwriten
>
>
> My daily 2c :-)
>
>
> /klacke

Hi list,

I have a few questions following up on Claes's comments. I'm still an  
Erlang newbie (hopefully not for too long), so please bear with me if  
they sound simple :)

- I'm a little confused about the 32 bit file offset issue. Integers  
in Erlang can be greater than 2^32, so how come dets offsets are  
limited to 32 bits? Is this a limitation of BEAM's native file IO code?
- Is dets limited to 32 bit offsets even if you run Erlang on 64 bit  
architectures?
- Is anybody planning on implementing Claes's solution to the repair  
time issue described above?

Your answers are appreciated!

Thanks,
Yariv