File sharing software

[Ulf Wiger (AL/EAB)]

Richard A. O'Keefe wrote:
> 
> Being a bear of very little brain, I thought this meant that while
> *dets* was limited to 2GB for a table, *mnesia* wasn't, and 
> when I look at the Mnesia manual, section 1.5.3 seems to say 
> that very thing:
> 
[...]
> 
> I must be getting senile.  Try as I might, I cannot make this 
> mean anything other than "mnesia can handle logical tables 
> bigger than 2GB".

You may not be getting senile just yet.
The limitation of dets is only relevant to mnesia in the case
of disc_only tables, and, as you observed, those tables can be
fragmented.

Ram copies are limited to the address space in erlang,
which is 4GB on 32-bit Erlang. On a 64-bit Erlang, one 
may may practical limitations before the address space
is exhausted. My own personal record in one ram-only 
mnesia table is about 15 GB of data. The machine I was
running on had only 16 GB of RAM, but there was no
significant performance degradation for a 15-GB table.

Yariv Sadan wrote:
> Yes, I did see that Mnesia can fragment Dets tables that must 
> grow over 2GB, but this solution just wasn't very appealing 
> to me because I'd rather use a storage engine that "just 
> works" without my having to manage table fragmentation as the 
> data grows. Fragmentation just sounded like it would be too 
> fragile and require too much maintenance, but then again I 
> don't have experience with Mnesia so I may very well be wrong 
> on this point.

I believe you are wrong on the maintenance part. There isn't 
really much maintenance at all. At some point, if your data
set keeps growing, you might want to add more fragments. This
is not at all difficult, and can be done in a running system.

I personally don't think that dets is an especially good
storage solution for very large databases. But in my 
formative years of learning database technology, the DBMS
would usually take a raw partition and even bypass the 
file system. In the 90's, this was the only way if you 
were serious about reliable and efficient database storage.
Perhaps this is not true anymore...? Obviously, MySQL
et al don't do it that way.

On the issue of binaries, nothing has changed since it was
last discussed on this list, AFAIK.

BR,
Ulf Wiger