Author: Rickard Green <rickard(at)erlang(dot)org>
Status: Accepted/24.0 Proposal is to be implemented in OTP release 24
Type: Standards Track
Erlang-Version: 24.0
Created: 01-Sept-2019
Post-History:

EEP 53: Process aliases preventing late replies reaching clients

Abstract

Currently there exists no lightweight mechanism for preventing late replies from a server to a client after a timeout or connection loss has occurred. The only way to prevent late replies today is to make the request via a proxy process.

The proposed process alias feature is a lightweight mechanism that solves the above problem. A process alias is similar to a registered name that is used temporarily while a request is outstanding. If the request times out or the connection to the server is lost, the alias is deactivated which prevents a late reply from reaching the client.

Copyright

This document has been placed in the public domain.

Specification

An alias is of the Erlang type reference() and can be used as destination when sending using the ! operator, or when sending using the erlang:send() and erlang:send_nosuspend() BIFs. An alias can be used both on local node and on remote nodes in a distributed system. The alias identifies a process that exist, or has existed, on the node with the node name returned by node(Alias).

All references will, as of this, be accepted as destination in the message send operations listed above. If the reference is not an alias or a previous alias that has been deactivated, the message will silently be dropped.

These new BIFs are introduced:

The spawn_opt() and spawn_request() BIFs have also been extended to accept an option {monitor, MonitorOpts} where MonitorOpts correspond to the option list of the monitor/3 BIF.

Full documentation of these BIFs and options can be found via pull request #2735 containing the reference implementation.

It is not possible to retrieve the process identifier of the process identified by an alias, and it is not possible to test if a reference is an alias or not.

Motivation

As previously stated it is possible to prevent late replies by using a proxy process that forwards the reply to the client. By spawning the proxy process and send its process identifier to the server instead of the clients own process identifier, the proxy can be terminated when the operation times out or the connection is lost. Since the proxy process is not alive, a reply will be silently dropped and no stray message will reach the previous client of the request. This however both makes the code more complicated and less efficient than it needs to be. The inefficiency comes from both the need to create, schedule, execute, and terminate the proxy process and the extra copying of data over the proxy process.

When the author of the client code has full control over the client process such late replies can be handled without a proxy since the code can be aware of these potential stray messages and drop them when received. This is, however, not possible when implementing library code. You then either need to use a proxy process, as done by the gen_statem behavior, or accept that the client process may get stray messages after a call, as done by the gen_server behavior.

Process aliases solves these issues with a very small overhead.

Rationale

Why use the reference data type for alias?

This is more or less what the reference data type is there for. A data type that can identify a huge amount of different entities. References are unique and contain a node identifier identifying the the node it originates from. This makes it easy to identify a specific process on a specific node while also identifying different aliases created by the same process. The embedded node identifier makes it easy to provide distribution transparency.

Why not make alias an opaque data type?

The expected most common use case is in a client server request. Such as gen_server:call(). Client server requests in Erlang are typically made while monitoring the server from the client. In order to minimize the data produced and sent in the request we want to reuse the reference created for identification of the monitor to also function as an alias. Since the monitor identifier is documented as a reference and is not opaque (which one can argue was a design mistake when introducing monitors), it becomes hard not to document the type of an alias as a reference as well.

Why not allow references as registered names in the already existing API?

There are two reasons. Distribution transparency and scalability.

Distribution transparency is really desirable since the user can use the functionality the same way regardless of whether it is a node local operation or node remote operation. The name registration API is not distribution transparent.

Regarding scalability. Due to how the name registration API has been designed we need some sort of table in order to implement the API. This table will be written to and read from by processes that are executing in parallel. In the use case we are focusing on, names (aliases) are expected to be temporary and created in huge amounts. That is, there will be large amounts of modifications of this table from processes executing on different processors. This will make it challenging to implement such a table that scales well.

In the proposed solution the information needed to route the message to the correct place is saved in the alias itself, the reference. The information needed to determine if the message passed via the alias should be dropped or passed along is saved in the process identified by the alias. That is, all information needed is distributed to where it is needed instead of being centralized in a node global table. This approach of distributed information introduce no new synchronization points at all when it has been fully implemented (more on that below) which will scale extremely well. An implementation based on a node global table can never compete scalability wise with that.

The already existing functionality for registered names cannot be implemented using this distributed information approach, but needs this centralized storage of names. That is, the already existing API cannot be used.

Besides node identifier a reference today contains three 32-bit words of data or in other words 96-bits of data. Of these 96 bits only 82 bits are allowed to be passed over the distribution to another node. This for historical reasons. While a reference resides locally it can however contain more or less unlimited amount of data. 82-bits are not enough to make a reference unique on the local node and at the same time uniquely identify a node local process. In order to be able to store all information needed in alias, the reference data type needs to be extended.

In the proposed solution references used as aliases are extended to use five 32-bit words on 64-bit architectures and four 32-bit words on 32-bit architectures. Since that much data in a reference cannot be passed over the distribution today, the reference implementation saves aliases that are alive in a node global table. When a node local alias enters the local node over the distribution one needs to look it up in this table in order to be able to restore it to its actual value. While aliases are passed around locally there is no need for look-ups in this table.

The reference implementation also modifies the distribution protocol to allow references with up to five 32-bit values. For backwards compatibility reasons this modification of the distribution protocol cannot be used at once when aliases are introduced. This since we need to be able to communicate with older nodes from previous releases for a while. When this has been living in the system for enough time (expected to be OTP 26) we can begin sending references with up to five 32-bit words and remove the usage of the table mapping references over the distribution to aliases. That is, it is not until this happens that the alias implementation is fully complete.

Why is it not possible to get the PID of the process that an alias refers to?

Most importantly there is no need to know the PID of the process that an alias refers to in order to solve the problems that alias are intended to solve. The user is expected to utilize alias in a protocol where one knows whether a reference is an alias or not and should not need to know the PID of the process that it refers to.

Besides the above there are also other issues with such functionality. The content of a reference is just a large integer. In order to keep distribution transparency one would either have to specify how this integer should be interpreted or require synchronous signaling with the node where the identified process resides. The synchronous signal-ling will be very expensive. By specifying how the reference integer should be interpreted we would prevent future changes to how the integer of the reference should be interpreted which might prevent future optimizations, improvements and new features. Up until the time when large references with five 32-bit words can be passed over the distribution, synchronous communication is also the only option on how to implement such functionality.

If we should mimic the whereis() function of the registered name API where you also can see if a name is currently registered, no other option than synchronous signaling with the process identified by the alias is possible.

Why is it not possible to test if a reference is an alias?

The same reason as to why it is not possible to get the PID of the process that is referred to by an alias.

Why not allow registration of arbitrary Erlang terms instead?

Such a feature could solve the same issue that aliases are intended to solve, but there are problems with such an approach.

Terms other than pids, ports, and references do not have a node identifier embedded into the data type. For such data types you need some other way to identify the node of where the name is registered. In the current case of atoms as registered names, this is done by wrapping the name in a two-tuple that contains the node name. Something like this is needed for all other terms than just plain pids, ports, and references. This also introduce a problem. Is a two-tuple just a name or a name plus a node identifier?

Should it be possible to register a PID as a name for another process? This would force all send operations to first lookup the PID in the table of registered names before performing the operation. This will cost performance in all send operations. The same is true for ports.

We don't think registration of arbitrary terms should be implemented due to the problems that arise. Current registration feature that only allows atoms can however be a bit too limiting when you need to register a number of processes for the same service. An option could be to allow registration of two-tuples containing an atom and an integer. Perhaps other terms such as strings should also be allowed, but arbitrary terms should not be allowed.

Allowing references as registered names implies scalability bottlenecks not present in the alias API. That is, this would be an inferior solution to the problem we set out to solve.

One probably wants to extend name registration with more allowed terms than just atoms, but this for solving other problems than what aliases are intended to solve. The name registration API does not fit aliases so we don't see that aliases should be combined with such an extension of the registration API. The alias solution solves the problem we set out to solve, so this eep is limited to that.

Why is the tag option of monitor/3 introduced?

When using the monitor option alias in a spawn_request() call you get unnecessary delays since you cannot share the alias with the child process until you have gotten the spawn reply with the process identifier of the child process. You instead typically want to explicitly create the alias before the spawn_request() call and pass it as an argument to the child process.

In a typical scenario you want to receive a response or an error of the operation. However, if you explicitly create an alias before the spawn_request() operation, the monitor reference and the alias will be different references. This will prevent the compiler from optimizing the receive (to skip messages present in the message queue when the reference was created) since not all receive clauses will match on the same reference.

We solve this by using the tag monitor option as well as the reply_tag spawn request. The following is a fully functional rpc implementation using this method on a system with the prototype implementation of aliases:

rpc(Node, M, F, A) ->
    Alias = alias([reply]),
    ReqId = spawn_request(Node,
                          fun () ->
                                  Result = apply(M, F, A),
                                  Alias ! {{result, Alias}, Result}
                          end,
                          [{monitor, [{tag, {'DOWN', Alias}}]},
                           {reply_tag, {spawn_reply, Alias}},
                           {reply, error_only}]),
    receive
        {{result, Alias}, Result} ->
            demonitor(ReqId, [flush]),
            Result;
        {{'DOWN', Alias}, ReqId, process, _, Error} ->
            rpc_error_cleanup(Alias, Error);
        {{spawn_reply, Alias}, ReqId, error, Error} ->
            rpc_error_cleanup(Alias, Error)
    end.

rpc_error_cleanup(Alias, Error) ->
    case unalias(Alias) of
        true ->
            %% No flush needed since we used the 'reply' option
            %% to alias(), and the alias was still active...
            error({rpc_error, Error});
        false ->
            %% Flush a possible result message...
            receive {{result, Alias}, Result} -> Result
            after 0 -> error({rpc_error, Error})
            end
    end.

The tag monitor option can be used in other situations as well in order to get a single reference that is present in all types of responses from a group of processes. The processes may be pre-existing or not. This reference can then be utilized to determine if a message corresponds to a specific operation made to a specific group of processes.

There are plans to extend the receive optimization so that multiple receives matching on the same reference in all clauses can utilize the optimization. This will also improve performance for such implementations receiving multiple messages matching on the same reference.

The tag to use in the monitor message is stored locally in the process that sets up the monitor and does not have to be communicated between processes. Most importantly it does not have to be sent over the wire in the distributed case. This also means that it can also be used when monitoring processes on older nodes which does not support this functionality.

Backwards Compatibility

The alias feature is a pure extension, so there are no real backwards compatibility issues.

In order to be able to communicate aliases over Erlang nodes from previous releases we cannot pass large references over the distribution and therefore need to keep information about aliases in a node global table. The implementation benefits from being able to pass larger references over the distribution, but will not do so until we can make it mandatory to be able to handle such large references. Both OTP 24 and OTP 25 will be able to handle large references over the distribution and since we only guarantee distribution compatibility with the two closest releases backwards and forwards we can then make large references mandatory in OTP 26.

This node global table for alias introduce an overhead when utilizing aliases compared to sending using the PID of the process. This due to allocation and manipulation of table structures. Comparing to the existing solution of utilizing a proxy process in order to prevent stray messages the overhead of this node global table for aliases is small. Fortunately this node global table also only need to be present temporarily and can be removed in OTP 26.

Reference Implementation

The reference implementation is provided by pull request #2735.

Beside implementation of the alias feature. The pull request also contain usage of aliases in the gen behaviors such as gen_server. Due to this it is now also possible to implement receive_response() functionality similar to erpc:receive_response() which also have been implemented:

Change Log