Mnesia and the Erlang message distribution model

[Ulf Wiger]

I would like to modify Dan's answer to say: yes, it is possible,
but you cannot rely entirely on the default behaviour of the OTP
components (wouldn't that have been sweet?)

On Thu, 21 Feb 2002, Kurt Luoto wrote:

>I would like to hear that it is possible to set up a sparse
>connection topology, a hierarchy of nodes where each node only
>needs to maintain connections with a few other nodes, for
>example, it's immediate "parent" or "master", it's immediate
>"children" or "slaves", and perhaps a few of it's "siblings" or
>"peers".

One thing you can do is to make the Erlang nodes on the line
cards (assuming you really want to run Erlang there) "hidden
nodes". This means that they will run distributed Erlang, but do
not appear in the nodes() list of the other nodes.

What to think about when using hidden nodes in particular, but
basically for distributed processing in general, is that you
should craft your APIs such that they hide the specifics of
inter-process(or) communication. Having done that, you could
replace distributed Erlang with a connection-less lightweight
protocol, with only minor changes to your code.

(If you want to get really sophisticated, you'll replace the
TCP/IP-based Distributed Erlang with your own model. This is not
for the faint of heart, but there's a user's guide that explains
how to do it.)

The scaling problems when running 200 to 300 Erlang nodes lie not
so much in Distributed Erlang, but in things like the global name
server. Using global_groups, one can partition the global name
space in order to make sure that global name registration doesn't
lead to voting among 300 peers.

Regarding mnesia, I don't think that it's terribly much overhead
to maintain the schema on 300 nodes, since the schema is static.
The trick is to allocate table copies in an efficient manner. You
could, for example, maintain table copies on two control
processors, for fault tolerance, and _maybe_ one local copy on a
line card, where the local processing occurs.

If you really need full replication of data, well, Mnesia is
probably as good as anything else (unless you come up with a
true multicast replication scheme of sorts.)


/Uffe