[erlang-questions] very large networks of erlang nodes?

[Richard O'Keefe]

On 15/02/2012, at 4:49 AM, Garrett Smith wrote:
> On Sat, Feb 11, 2012 at 4:53 PM, Miles Fidelman
> <mfidelman@REDACTED> wrote:
>> Does anybody have experience running an Erlang environment consisting of
>> 100s, 1000s, or more nodes, spread across a network?
> As has been said, the fully connected mesh practically limits you to
> 50 - 150 nodes, at least in my experience. You will also find that
> connections flap quite a bit across unreliable networks, wreaking
> havoc in your application if you don't design for it.

What would happen if you had 1000 nodes in a box with a reliable but
not ultrafast interconnect?  I'm not talking about multicore here,
although 16 Tileras in a smallish box doesn't seem unlikely any more,
but say 1000 separate-physical-address-space nodes connected as a
tree or a hypercube or something.

Could distributed Erlang be set up in some hierarchical fashion?

It seems to me that there are three issues:
 - number of points of authentication
	(network: many; cluster-in-a-box: one)
 - number of eavesdropping points
	(network: many; cluster-in-a-box: one)
 - number of communicating devices
	(network: many; cluster-in-a-box: many)
and that just thinking in terms of authentication and eavesdropping,
distributed Erlang makes perfect sense for cluster-in-a-box,
IF it works at that scale.

The Magnus project that Fergus O'Brien was involved with would have
been using Erlang in this way, I believe.