[erlang-questions] Erlang cluster over a WAN link?

[Francis Joanis]

Hi,

I haven't had a lot of experience with actual Erlang clusters
separated between a WAN, but if you are only talking about 2 nodes
then I suspect the actual cluster connection will be almost identical
to using vanilla TCP connection.

Once connected, the distributed nodes will use the Erlang Distribution
Protocol which is built on top of TCP. If you were to use a plain TCP
socket you would still need to come up with some kind of protocol to
stream data over the connection. Chances are that this protocol will
look a lot like the built-in Erlang Distribution Protocol (especially
if you use term_to_binary/binary_to_term). One difference that the EDP
will have is a requirement to work with epmd and this might cause you
some more headaches in terms of firewall configuration (to make the
epmd port + the dynamically allocated EDP port available).

It depends on the type of application you are dealing with. If you
need deterministic/real time behavior then maybe TCP and/or WAN isn't
a good choice to start with (TCP will try to throttle connections to
avoid congestion, etc., so you might end up with delays being added at
the TCP level). UDP could have a more real time behavior but you would
need to implement your own reliability/congestion management layer on
top of it (or use an alternative transport).

If you can tolerate these delays, I'd say try it and see how it flies.
There are tools available that can let you simulate
delays/bandwidth/packet loss so that you can try it using a few VMs
without an actual WAN connection. One of those is dummynet (available
on FreeBSD and others).

With the EDP, if you need more than two nodes, you could control how
the "meshing" of the nodes is done using the VM "-connect_all false"
flag (otherwise it would try to create a lot of TCP connections to
form a mesh). There is also this flag that would apply to control the
"heartbeating" of the nodes: net_ticktime
(http://erlang.org/doc/man/kernel_app.html).

You might also find these useful:

https://erlangcentral.org/wiki/index.php?title=Distributed_erlang_using_ssl_through_firewalls
http://fjoanis.github.io/2013/08/31/Network_Simulation_FreeBSD_DummyNet/

Salut,
Francis

On Mon, Oct 7, 2013 at 4:13 AM, Alexandre Beaulne
<alexandre.beaulne@REDACTED> wrote:
> Hello everyone,
>
> I am working on two tightly coupled (but physically separated) Erlang apps
> communicating between each other by TCP over a WAN link. I was curious why
> an Erlang cluster wasn't used in this case and searched around a bit with no
> clear cut answer. I read that it's generally a bad idea to set up a RabbitMQ
> cluster using WAN instead of LAN, but if I understand well it has to do with
> the fact that RabbitMQ lies on the Availability/Partition-tolerance axis of
> the CAP theorem while in our case we'd be more on the
> Consistency/Partition-tolerance axis so it might make it more suitable for a
> cluster? The WAN in question is a leased line that has been solid
> historically and the number of nodes will probably stay at 2 in the
> foreseeable future.
>
> Anyone here has successfully run an Erlang cluster over a WAN link?
>
> Is there good reasons to use a plain vanilla TCP connection rather than an
> Erlang cluster in this use case?
>
> It is sad to have a nice, concurrent Erlang application having to funnel
> into a single, sequential "TCP" process only to re-branch on the other side
> of the connection...
>
> Thank you very much for you time,
>
> Alexandre Beaulne
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>