[erlang-patches] Added net_kernel support for multiple transport protocols

[Serge Aleynikov]

Hi,

[Note: this patch is submitted by email as as a pull request #121]

Short problem description
=========================
  I want an Erlang node to listen on incoming connections using
  TCPv4, TCPv6, UDS, and TLS transports on different ports.

  I'd like other nodes to connect to this node using TLS if nodes
  are in external networks, use TCPv4 or TCPv6 if nodes are
  in the same subnet, and use UDS if nodes are on the same machine.

  Present implementation requires that all nodes in the network
  use the same transport, which is a huge limitation

git fetch git://github.com/saleyn/otp.git proto_dist

https://github.com/saleyn/otp/compare/erlang:master...proto_dist
https://github.com/saleyn/otp/compare/erlang:maint...proto_dist.patch

Long solution description
=========================

This patch addresses the limitation of registering only one Erlang
distributed transport with Erlang Port Mapping Daemon, and therefore
enabling a node to listen for distributed node connections on multiple
port using different protocols, and selectively choose the protocol
to use when connecting to a node from different secure and unsecure
networks.

The legacy implementation of Erlang distributed transport allows to
start a node with more than one transport via the
"-proto_dist mod1 mod2 ..." command-line option. But epmd didn't support
registration of multiple protocols. And net_kernel doesn't implement a
way of figuring out which transports are supported on a remote node and
doesn't allow to select a desired transport protocol of choice to use it
for connecting to the node.

The current patch extends epmd by having it store the name of the
protocol handling module passed to the -proto_dist startup option during
node registration.  Multiple registrations are allowed, and they can be
registered to different ports.  This way a node can start distributed
transport listeners (e.g. TCP, SSL, UDS), and register them with epmd.

When another node does epmd "PORT_PLEASE" lookup in order to connect to
node X, it now gives to epmd a list of transport names it supports (i.e.
the ones that were passed to it via -proto_dist option at startup)
together with the node name it queries.  The epmd filters out only the
transports supported by the calling client node, and gives back only
those along with the corresponding port numbers.

net_kernel now supports a new command-line option
"-proto_dist_mf Mod Fun" which is a user-define callback function that
can be used to resolve which transport protocol to use to connect to
a node. This function is called after the epmd query returned a list
of protocols supported by the target node. By default the first
transport in the list is used. An example implementation of this
function is included in
`lib/kernel/examples/uds_dist/src/dist_selector.erl`.

In order to support backward compatibility the EPMD protocol was
extended with new commands documented in erl_dist_rotocol.xml. Old
nodes can query the new EPMD version, but new nodes (using this patch)
require new version of EPMD.

Here's an illustration:

  # Node 's' is started with TCP and TLS support
  # see http://www.erlang.org/doc/apps/ssl/ssl_distribution.html#id61752
  # note that lib/kernel/examples/uds_dist/src/dist_selector.erl
  # module is used to do custom transport selection

  $ erl -boot start_ssl -proto_dist inet_tcp inet_tls -ssl_difile
"/home/serge/tmp/key.pem" -ssl_dist_opt server_certfile
"/home/serge/tmp/certificate.pem" -ssl_dist_opt
server_secure_renegotiate true client_secure_renegotiate true
-proto_dist_mf dist_selector select -pa
"lib/kernel/examples/uds_dist/ebin" -sname s@REDACTED

  (s@REDACTED)1> erl_epmd:names().
  [{ok,{"s",[{49645,"inet_tcp"},{41477,"inet_tls"}]}}]
  (s@REDACTED)2>

As we see the node registered itself with epmd and has TCP listener on
port 49645, and TLS listener on port 41477.

We now start another node with just TCP transport:

  $ install/bin/erl -sname k@REDACTED
  Erlang R17A (erts-5.11) [source-7e9f18c] [64-bit] [smp:4:4]
   [async-threads:10] [hipe] [kernel-poll:false]

  Eshell V5.11  (abort with ^G)
  (k@REDACTED)1> erl_epmd:names().
  [{ok,{"k",[{38558,"inet_tcp"}]}},
   {ok,{"s",[{49645,"inet_tcp"},{41477,"inet_tls"}]}}]
  (k@REDACTED)2>

Connect it to node 's':

  (k@REDACTED)2> net_kernel:connect_node(s@REDACTED).
  true
  (k@REDACTED)4> nodes().
  [s@REDACTED]

Start the 3rd node 'm' with TCP and SSL, and connect it to node 's':

  $ erl -boot start_ssl -proto_dist inet_tcp inet_tls -ssl_difile
"/home/serge/tmp/key.pem" -ssl_dist_opt server_certfile
"/home/serge/tmp/certificate.pem" -ssl_dist_opt
server_secure_renegotiate true client_secure_renegotiate true
-proto_dist_mf dist_selector select -pa
"/home/serge/tmp/otp/lib/kernel/examples/uds_dist/ebin" -sname m@REDACTED

  % Let's query the EPMD to see which transports/ports are available
  % to connect
  (m@REDACTED)1>
erl_epmd:port_please(s,"localhost",net_kernel:dist_protos(),infinity).
  {ports,[{49645,"inet_tcp"},{41477,"inet_tls"}],
         5,
         [{epmd,{{127,0,0,1},4369}}]}

  % Let the magic begin (since this is a node in the same subnet,
  % dist_selector will select the TCP protocol

  (m@REDACTED)2> net_kernel:connect_node(s@REDACTED).
  true

This is a long email, and a sizable patch.  I hope others find it useful
and it can be included in the distribution.

Regards,

Serge