[erlang-questions] very large networks of erlang nodes?

[Garrett Smith]

On Sat, Feb 11, 2012 at 4:53 PM, Miles Fidelman
<mfidelman@REDACTED> wrote:
> Does anybody have experience running an Erlang environment consisting of
> 100s, 1000s, or more nodes, spread across a network?
>
> I'm thinking about applications like SETI@REDACTED and folding@REDACTED, that
> distribute processing tasks across huge numbers of PCs, taking advantage of
> shared cycles to run long-running, data intensive jobs that can be
> parallelized -- and more generally about the BOINC platform
> (http://boinc.berkeley.edu).
>
> Seems to me that Erlang would be a great platform for such things.  Sort of
> wondering if anybody has played with anything along such lines.

As has been said, the fully connected mesh practically limits you to
50 - 150 nodes, at least in my experience. You will also find that
connections flap quite a bit across unreliable networks, wreaking
havoc in your application if you don't design for it.

We (CloudBees) have several hundred Erlang VMs running in
non-distributed mode. Our main backbone for messaging is RabbitMQ, but
we're actively moving away from AMQP toward 0MQ.

The cost for moving outside distributed Erlang is you lose the famed
"location transparency" feature that lets you seamlessly distribute
your application without significant changes to your code. The upside
is you can safely move well beyond the limits of a full connected
mesh.

I haven't tried with the hierarchical / hidden node approach that
Adrian mentioned, but I'm very glad to see a number like 750!

If you're talking SETI like applications, that suggests an unbounded
limit. I'd definitely look at 0MQ -- you'll have enough low level
flexibility in your messaging topology to tackle that hard problem.

Garrett