[erlang-questions] Re: how to scale into the cloud using process? example computing simple average

[Ulf Wiger]

----- "Joel Reymont" <joelr1@REDACTED> wrote:

> The intractable (I think) problem is using Mnesia as a backend for  
> internet services. I don't think it can be used since there's now way 
> 
> to automatically sync up the various database instances after a  
> network split.

I think it ought to be fairly straightforward to design
an arbitrator similar to the one used by MySQL Cluster.
The tools needed to detect partitioned network and to 
resolve it are there - the thing that mnesia doesn't 
provide is a default resolution algorithm. If the 
algorithm used by MySQL Cluster is acceptable, I see
no reason why it couldn't be applied to Mnesia as well.

(For those who don't want to go chasing for it,
the MySQL Arbitrator basically selects the biggest
group of connected nodes, or the group that reported
to the Arbitrator last. This is from memory. The details
may be wrong. 

FWIW, the AXD 301 used a somewhat similar arbitration 
mechanism: each node tried to ping itself through the 
switch core. If it couldn't, it had no means to control 
the HW resources and had to reboot; otherwise, the 
node that had a stable copy of the traffic handling
application became the master; if both nodes were 
running traffic handling (which could happen if 
the outage lasted a few seconds), I believe the one 
hard-wired as Control Processor 1 was designated 
master.)

Bottom line: if you can pick an arbitration algorithm
that offers suitable recovery behaviour, I believe you
can implement it on top of what mnesia provides.
One could imagine a peer-to-peer resolution protocol
(difficult to make generic) or a 'central observer',
which is the MySQL strategy.

Or are you referring to the lack of a 'merge' of 
two table copies? This is a more difficult thing to 
add. Right now, that would have to be done by 
extracting records from the non-master, have it 
restart and load tables from the master, then 
re-inserting the records. Since this cannot be done
within a larger transaction, it might be necessary to 
delay requests from the application layer during this
time.

Which DBMSes have a good solution to this problem?
How do they do it? And what prevents using the same
strategy with mnesia?

(Mnesia's supposed lack of support for handling 
partitioned networks has more or less become an
urban legend. Everyone assumes that this is a 
major weakness in mnesia, but there is very little
detail about how the competing alternatives 
supposedly handle this in a much better way.
I would love to be enlightened about this.)

BR,
Ulf W
-- 
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com