[erlang-questions] very large networks of erlang nodes?

[Michael Truog]

On 02/11/2012 02:53 PM, Miles Fidelman wrote:
> Does anybody have experience running an Erlang environment consisting of 100s, 1000s, or more nodes, spread across a network?
>
> I'm thinking about applications like SETI@REDACTED and folding@REDACTED, that distribute processing tasks across huge numbers of PCs, taking advantage of shared cycles to run long-running, data intensive jobs that can be parallelized -- and more generally about the BOINC platform (http://boinc.berkeley.edu).
>
> Seems to me that Erlang would be a great platform for such things.  Sort of wondering if anybody has played with anything along such lines.
>
Conceptually, Erlang would be a wonderful fit.  However, distributed Erlang creates a fully connected network topology which is a limiting factor, that limits a distributed Erlang cluster to 50 - 100 nodes roughly.  I do not have proof from trying this on my own, just knowledge of the attempts of others.  The net tick time within distributed Erlang expects a response every minute with some permissible variation.  You can try changing the net tick time, however, I have seen breakage when using links/monitors between distributed Erlang nodes with a net tick time set longer than 60 seconds (so I assume the net tick time was hard-coded internally, in some places).  You can always try doing something strange with hidden nodes trying to limit the scope of a fully-connected network, but it seems like more trouble than it is worth.

I have a project called CloudI (http://cloudi.org) that provides a SOA framework that allows you to connect CloudI distributed Erlang clusters together with ZeroMQ bridges, to surpass the distributed Erlang fully connected network topology limitations.  When doing things that way, you would create CloudI services that communicated through message passing using the CloudI API, and the internal routing takes care of service communication (through ZeroMQ or distributed Erlang messaging).  Its an open source project (BSD license), so you host it yourself "for greater good".

- Michael