[erlang-questions] MySQL cluster - MnesiaXXL(tm) :)

[Ulf Wiger]

Den 2006-10-21 16:32:34 skrev Pat e <patrickerj@REDACTED>:

> How much work would it be needed to tweak Mnesia for massive database  
> storing?

I think more than tweaking is involved, as Jay pointed out.
You may, for example, read the following document from a
vendor that claims to manage database storage in the petabyte
range:

http://www.teradata.com/t/pdf.aspx?a=83673&b=84876

(They may even have some patents that will get in the way
of such tweaking...)


> Will it be necessary to completley rewrite mnesia for
> multi-terabyte disk-only tables?

Quite possibly, but this seems like an unrealistic undertaking.
OTOH, developing pieces that would bring erlang closer to
supporting multi-gigabyte disk storage solutions might be
of value.

Then again, the larger the data volume, the higher the price
of failure (most likely). There are products that can do this
today. They are usually quite expensive, but then again, most
people who juggle terabytes of data are prepared to pay a pretty
large sum of money for solid products and support. It's a
difficult problem, to say the least.

I would assume that if anyone would develop such a beast in
Erlang, they wouldn't put it up on SourceForge, but sell it
for big bucks. ;)


> Has anyone have any idea how to bypass 2GB limit for
> disk only tables, and how to effecivlly operate ram caching?


You may google on 'site:erlang.org dets limit' and find several posts on  
the list on this topic. Here are two:

http://www.erlang.org/ml-archive/erlang-questions/200004/msg00077.html

http://www.erlang.org/ml-archive/erlang-questions/200604/msg00264.html

As Håkan Mattsson's post (first one above) points out, ram caching
would only solve one problem. Remote table load remains to be
solved as well.


> I think we will need to develop some blueprint in next years of
> how to develop such powerful upgrade for Mnesia.
>
> Ulf... again, can you help? :)

Not much, I think. I prototyped a low-level interface,
as part of the rdbms patches, that would allow for plugging in
other table types in mnesia. I did toy with the idea of trying
to use it for a caching disk-only table type, but haven't even
started an implementation for it. I'm not completely convinced
that the (external_copies) approach I took was the right one,
and (more importantly), neither is Dan Gudmundsson, who
maintains mnesia.

And right now, I have lots of other things that get higher
priority.

BR,
Ulf W
-- 
Ulf Wiger