File sharing software

[Yariv Sadan]

On May 3, 2006, at 5:43 AM, Ulf Wiger (AL/EAB) wrote:

> Richard A. O'Keefe wrote:
>>
>> Being a bear of very little brain, I thought this meant that while
>> *dets* was limited to 2GB for a table, *mnesia* wasn't, and
>> when I look at the Mnesia manual, section 1.5.3 seems to say
>> that very thing:
>>
> [...]
>>
>> I must be getting senile.  Try as I might, I cannot make this
>> mean anything other than "mnesia can handle logical tables
>> bigger than 2GB".
>
> You may not be getting senile just yet.
> The limitation of dets is only relevant to mnesia in the case
> of disc_only tables, and, as you observed, those tables can be
> fragmented.
>
> Ram copies are limited to the address space in erlang,
> which is 4GB on 32-bit Erlang. On a 64-bit Erlang, one
> may may practical limitations before the address space
> is exhausted. My own personal record in one ram-only
> mnesia table is about 15 GB of data. The machine I was
> running on had only 16 GB of RAM, but there was no
> significant performance degradation for a 15-GB table.
>
> Yariv Sadan wrote:
>> Yes, I did see that Mnesia can fragment Dets tables that must
>> grow over 2GB, but this solution just wasn't very appealing
>> to me because I'd rather use a storage engine that "just
>> works" without my having to manage table fragmentation as the
>> data grows. Fragmentation just sounded like it would be too
>> fragile and require too much maintenance, but then again I
>> don't have experience with Mnesia so I may very well be wrong
>> on this point.
>
> I believe you are wrong on the maintenance part. There isn't
> really much maintenance at all. At some point, if your data
> set keeps growing, you might want to add more fragments. This
> is not at all difficult, and can be done in a running system.
>
> I personally don't think that dets is an especially good
> storage solution for very large databases. But in my
> formative years of learning database technology, the DBMS
> would usually take a raw partition and even bypass the
> file system. In the 90's, this was the only way if you
> were serious about reliable and efficient database storage.
> Perhaps this is not true anymore...? Obviously, MySQL
> et al don't do it that way.
>
> On the issue of binaries, nothing has changed since it was
> last discussed on this list, AFAIK.
>
> BR,
> Ulf Wiger

Maybe it's possible to develop for Mnesia an alternative disc only  
storage engine designed for storing large blobs. It wouldn't have to  
be an Oracle killer. It would just have to handle very large data  
sets, have decent performance, and avoid the rebuilding cost  
associated with broken dets tables. Such a solution would make Mnesia  
more versatile and therefore more appealing for applications for  
which it makes sense to sacrifice soft real-time performance for  
large storage capacity. What do you think?

Regards,
Yariv