[erlang-questions] Description of Erlang's distribution mechanisms.

[Cian Synnott]

  http://emauton.org/disterl/

I wrote this a year ago while digging around in Erlang's distribution
mechanisms; I had intended to finish it (in particular, to provide
some notes on performance characteristics, what blocks where, etc.)
but it seems to have never quite surfaced to the top of my Copious
Free Time list again. :o) Lukas' message about the GC description
reminded me, and it's probably better to have something referenced on
the list here (even unfinished) than for another engineer to do the
same investigation.

If anyone has any comments on accuracy or things that should be
included, let me know. That said, I have a very good set of comments
from Evan Vigil McClanahan that require a bit more work to integrate.
One of these days. :o) Included below for competeness:

---- 8< ----
One thing I would love to have documented somewhere
publicly are the performance/blocking semantics of sending messages
over disterl (which is something that I haven't looked into myself,
although it was on my long list of perf issues to look into before).
What I mean here is: how is the message serialized when
you send it?  Does serialization block the sending pid or is it
asynchronous?  Are there differences between sending something over
gen_tcp vs. disterl, etc.  I have intuitions about all of these
things, but it'd be nice to know.  It might require some beam
spelunking to figure those out though.

Other things:

- disterl does some work to make sure that messages arrive in order
- it doesn't multiplex over multiple sockets, and is often subject to
severe head-of-line blocking issues when you send large messages over
it because of that.
- please document the +zddbl argument and busy_dist_ports, which are
important to understand and tune if you're ever sending lots of data
over disterl pipes.
- nodeup and nodedown messages ought to be covered
- for fast sending there are some issues with the socket buffer
resizing logic that can limit speed in the case of many small
messages, setting the disterl socket buffers larger can help
- often people want to dynamically or selectively startup the
net_kernel from an escript or something, a quick recipe might help
- a short discussion of net_ticktime and dead node detection might be
helpful, as the default is quite high for some applications and
migrating later can be quite painful

---- 8< ----

Cian