Erlang logo
User's Guide
Reference Manual
Release Notes
PDF
Top

Erlang Run-Time System Application (ERTS)
User's Guide
Version 5.9.1


Expand All
Contract All

Chapters

9 Distribution Protocol

The description here is far from complete and will therefore be further refined in upcoming releases. The protocols both from Erlang nodes towards EPMD (Erlang Port Mapper Daemon) and between Erlang nodes, however, are stable since many years.

The distribution protocol can be divided into four (4) parts:

  • 1. Low level socket connection.

  • 2. Handshake, interchange node name and authenticate.
  • 3. Authentication (done by net_kernel).
  • 4. Connected.

A node fetches the Port number of another node through the EPMD (at the other host) in order to initiate a connection request.

For each host where a distributed Erlang node is running there should also be an EPMD running. The EPMD can be started explicitly or automatically as a result of the Erlang node startup.

By default EPMD listens on port 4369.

3 and 4 are performed at the same level but the net_kernel disconnects the other node if it communicates using an invalid cookie (after one (1) second).

The integers in all multi-byte fields are in big-endian order.

9.1  EPMD Protocol

The requests served by the EPMD (Erlang Port Mapper Daemon) are summarized in the figure below.

IMAGE MISSING
Figure 9.1:   Summary of EPMD requests.

Each request *_REQ is preceded by a two-byte length field. Thus, the overall request format is:

2 n
Length Request
Table 9.1:  

Register a node in the EPMD

When a distributed node is started it registers itself in EPMD. The message ALIVE2_REQ described below is sent from the node towards EPMD. The response from EPMD is ALIVE2_RESP.

1 2 1 1 2 2 2 Nlen 2 Elen
120 PortNo NodeType Protocol HighestVersion LowestVersion Nlen NodeName Elen Extra
Table 9.2:   ALIVE2_REQ (120)
PortNo
The port number on which the node accept connection requests.
NodeType
77 = normal Erlang node, 72 = hidden node (C-node),...
Protocol
0 = tcp/ip-v4, ...
HighestVersion
The highest distribution version that this node can handle. The value in R6B and later is 5.
LowestVersion
The lowest distribution version that this node can handle. The value in R6B and later is 5.
Nlen
The length of the NodeName.
NodeName
The NodeName as a string of length Nlen.
Elen
The length of the Extra field.
Extra
Extra field of Elen bytes.

The connection created to the EPMD must be kept as long as the node is a distributed node. When the connection is closed the node is automatically unregistered from the EPMD.

The response message ALIVE2_RESP is described below.

1 1 2
121 Result Creation
Table 9.3:   ALIVE2_RESP (121)

Result = 0 -> ok, Result > 0 -> error

Unregister a node from the EPMD

A node unregisters itself from the EPMD by simply closing the TCP connection towards EPMD established when the node was registered.

Get the distribution port of another node

When one node wants to connect to another node it starts with a PORT_PLEASE2_REQ request towards EPMD on the host where the node resides in order to get the distribution port that the node listens to.

1 N
122 NodeName
Table 9.4:   PORT_PLEASE2_REQ (122)

where N = Length - 1

1 1
119 Result
Table 9.5:   PORT2_RESP (119) response indicating error, Result > 0.

Or

1 1 2 1 1 2 2 2 Nlen 2 Elen
119 Result PortNo NodeType Protocol HighestVersion LowestVersion Nlen NodeName Elen Extra
Table 9.6:   PORT2_RESP when Result = 0.

If Result > 0, the packet only consists of [119, Result].

EPMD will close the socket as soon as it has sent the information.

Get all registered names from EPMD

This request is used via the Erlang function net_adm:names/1,2. A TCP connection is opened towards EPMD and this request is sent.

1
110
Table 9.7:   NAMES_REQ (110)

The response for a NAMES_REQ looks like this:

4  
EPMDPortNo NodeInfo*
Table 9.8:   NAMES_RESP

NodeInfo is a string written for each active node. When all NodeInfo has been written the connection is closed by EPMD.

NodeInfo is, as expressed in Erlang:

	io:format("name ~s at port ~p~n", [NodeName, Port]).
	

Dump all data from EPMD

This request is not really used, it should be regarded as a debug feature.

1
100
Table 9.9:   DUMP_REQ

The response for a DUMP_REQ looks like this:

4  
EPMDPortNo NodeInfo*
Table 9.10:   DUMP_RESP

NodeInfo is a string written for each node kept in EPMD. When all NodeInfo has been written the connection is closed by EPMD.

NodeInfo is, as expressed in Erlang:

	io:format("active name     ~s at port ~p, fd = ~p ~n",
	          [NodeName, Port, Fd]).
	

or

	io:format("old/unused name ~s at port ~p, fd = ~p~n",
	          [NodeName, Port, Fd]).
	

Kill the EPMD

This request will kill the running EPMD. It is almost never used.

1
107
Table 9.11:   KILL_REQ

The response fo a KILL_REQ looks like this:

2
OKString
Table 9.12:   KILL_RESP

where OKString is "OK".

STOP_REQ (Not Used)

1 n
115 NodeName
Table 9.13:   STOP_REQ

where n = Length - 1

The current implementation of Erlang does not care if the connection to the EPMD is broken.

The response for a STOP_REQ looks like this.

7
OKString
Table 9.14:   STOP_RESP

where OKString is "STOPPED".

A negative response can look like this.

7
NOKString
Table 9.15:   STOP_NOTOK_RESP

where NOKString is "NOEXIST".

9.2  Handshake

The handshake is discussed in detail in the internal documentation for the kernel (Erlang) application.

9.3  Protocol between connected nodes

As of erts version 5.7.2 the runtime system passes a distribution flag in the handshake stage that enables the use of a distribution header on all messages passed. Messages passed between nodes are in this case on the following format:

4 d n m
Length DistributionHeader ControlMessage Message
Table 9.16:  

where:

Length is equal to d + n + m

ControlMessage is a tuple passed using the external format of Erlang.

Message is the message sent to another node using the '!' (in external format). Note that Message is only passed in combination with a ControlMessage encoding a send ('!').

Also note that the version number is omitted from the terms that follow a distribution header.

Nodes with an erts version less than 5.7.2 does not pass the distribution flag that enables the distribution header. Messages passed between nodes are in this case on the following format:

4 1 n m
Length Type ControlMessage Message
Table 9.17:  

where:

Length is equal to 1 + n + m

Type is: 112 (pass through)

ControlMessage is a tuple passed using the external format of Erlang.

Message is the message sent to another node using the '!' (in external format). Note that Message is only passed in combination with a ControlMessage encoding a send ('!').

The ControlMessage is a tuple, where the first element indicates which distributed operation it encodes.

LINK

{1, FromPid, ToPid}

SEND

{2, Cookie, ToPid}

Note followed by Message

EXIT

{3, FromPid, ToPid, Reason}

UNLINK

{4, FromPid, ToPid}

NODE_LINK

{5}

REG_SEND

{6, FromPid, Cookie, ToName}

Note followed by Message

GROUP_LEADER

{7, FromPid, ToPid}

EXIT2

{8, FromPid, ToPid, Reason}

9.4  New Ctrlmessages for distrvsn = 1 (OTP R4)

SEND_TT

{12, Cookie, ToPid, TraceToken}

Note followed by Message

EXIT_TT

{13, FromPid, ToPid, TraceToken, Reason}

REG_SEND_TT

{16, FromPid, Cookie, ToName, TraceToken}

Note followed by Message

EXIT2_TT

{18, FromPid, ToPid, TraceToken, Reason}

9.5  New Ctrlmessages for distrvsn = 2

distrvsn 2 was never used.

9.6  New Ctrlmessages for distrvsn = 3 (OTP R5C)

None, but the version number was increased anyway.

9.7  New Ctrlmessages for distrvsn = 4 (OTP R6)

These are only recognized by Erlang nodes, not by hidden nodes.

MONITOR_P

{19, FromPid, ToProc, Ref} FromPid = monitoring process ToProc = monitored process pid or name (atom)

DEMONITOR_P

{20, FromPid, ToProc, Ref} We include the FromPid just in case we want to trace this. FromPid = monitoring process ToProc = monitored process pid or name (atom)

MONITOR_P_EXIT

{21, FromProc, ToPid, Ref, Reason} FromProc = monitored process pid or name (atom) ToPid = monitoring process Reason = exit reason for the monitored process