File sharing software
Wed May 3 17:38:14 CEST 2006
On May 3, 2006, at 5:43 AM, Ulf Wiger (AL/EAB) wrote:
> Richard A. O'Keefe wrote:
>> Being a bear of very little brain, I thought this meant that while
>> *dets* was limited to 2GB for a table, *mnesia* wasn't, and
>> when I look at the Mnesia manual, section 1.5.3 seems to say
>> that very thing:
>> I must be getting senile. Try as I might, I cannot make this
>> mean anything other than "mnesia can handle logical tables
>> bigger than 2GB".
> You may not be getting senile just yet.
> The limitation of dets is only relevant to mnesia in the case
> of disc_only tables, and, as you observed, those tables can be
> Ram copies are limited to the address space in erlang,
> which is 4GB on 32-bit Erlang. On a 64-bit Erlang, one
> may may practical limitations before the address space
> is exhausted. My own personal record in one ram-only
> mnesia table is about 15 GB of data. The machine I was
> running on had only 16 GB of RAM, but there was no
> significant performance degradation for a 15-GB table.
> Yariv Sadan wrote:
>> Yes, I did see that Mnesia can fragment Dets tables that must
>> grow over 2GB, but this solution just wasn't very appealing
>> to me because I'd rather use a storage engine that "just
>> works" without my having to manage table fragmentation as the
>> data grows. Fragmentation just sounded like it would be too
>> fragile and require too much maintenance, but then again I
>> don't have experience with Mnesia so I may very well be wrong
>> on this point.
> I believe you are wrong on the maintenance part. There isn't
> really much maintenance at all. At some point, if your data
> set keeps growing, you might want to add more fragments. This
> is not at all difficult, and can be done in a running system.
> I personally don't think that dets is an especially good
> storage solution for very large databases. But in my
> formative years of learning database technology, the DBMS
> would usually take a raw partition and even bypass the
> file system. In the 90's, this was the only way if you
> were serious about reliable and efficient database storage.
> Perhaps this is not true anymore...? Obviously, MySQL
> et al don't do it that way.
> On the issue of binaries, nothing has changed since it was
> last discussed on this list, AFAIK.
> Ulf Wiger
Maybe it's possible to develop for Mnesia an alternative disc only
storage engine designed for storing large blobs. It wouldn't have to
be an Oracle killer. It would just have to handle very large data
sets, have decent performance, and avoid the rebuilding cost
associated with broken dets tables. Such a solution would make Mnesia
more versatile and therefore more appealing for applications for
which it makes sense to sacrifice soft real-time performance for
large storage capacity. What do you think?
More information about the erlang-questions