3 Supervision Principles

This section introduces a process structuring model for programming fault tolerant applications. The model is based on the idea of workers, supervisors, and a supervision tree.

• Workers are processes which perform computations.
• Supervisors are processes which monitor the behaviour of workers. A supervisor can restart a worker if something goes wrong.
• The supervision tree is a hierarchical arrangement of an application into workers and supervisors.

A supervisor can supervise either workers or supervisors. Hereafter we will use the term child to mean either a worker or a supervisor.

Supervision trees can be manipulated by using the functions exported from the module supervisor.

A typical supervision tree looks as follows:

Supervision Tree

• The square boxes represent supervisors.
• The circles represent workers.
• The supervisors are annotated with either "1" (one_for_one), or "a" (one_for_all). This denotes the type of supervision strategy used.

One_for_one Supervision

One_for_one Supervision. If a the supervised child dies, then this child only is restarted according to the start specification.

All_for_one Supervision

One_for_all Supervision. If a supervised child dies, then all the supervised children will be terminated according to the shutdown specification and all will then be restarted according to the start specification.

One_for_one supervision is often used when the processes supervised are independent of each other. If the processes are independent, they can often be programmed as sets of processes with no internal communication between them.

One_for_all supervision is often used when the processes supervised depend on each other. If the processes are dependent, they can often be programmed as sets of synchronized, communicating processes.

3.1 The Supervision Tree

Supervision trees are created by using an instance of the supervisor behaviour. A supervision tree is created from a supervision start specification. The module xx is an example of the supervision behaviour. xx:init(Args) must be written so as to return {ok, SuperSpec}, or {error, What}, or ignore. The supervision tree is created with the following call:

supervisor:start_link(Name, xx, Args)

Name is the name of the supervision tree and can be used to manipulate the supervision tree. Args is an argument which is passed to xx:init. Name can be omitted if an anonymous supervisor is started.

-module(xx).
-behaviour(supervisor).
-export([init/1]).
init(_) ->
    {ok,{{one_for_all, MaxR, MaxT},
         [
          {log,   {log, start, [25]}, permanent, 5000,     worker,     [log]},
          {mysup, {mysup, start, []}, permanent, infinity, supervisor, [mysup]},
         ...
          {ChildName, MFA,            Type,      Shutdown, Class,      Modules}
        ]}}.

• xx:init returns a tuple of the form {ok, SupSpec}.
• SupSpec is a term of the form {SupFlags, [ChildSpec]}.
• SupFlags specifies the supervision to be of type one_for_one or one_for_all, and it specifies the restart frequency.
• ChildSpec is a list of specifications (one per child) which specifies how each child should be started and treated.

Each child can be one of three types:

• a permanent child is always restarted when it dies
• a transient child is restarted if it dies abnormally
• a temporary child is never restarted.

All child processes which are added to the supervision tree must implement the shutdown protocol correctly (refer to the section The Shutdown Protocol). All the generic servers, event handlers, and other standard system behaviours automatically implement this protocol.

User processes which are implemented without using the standard behaviours must follow the shutdown protocol. This protocol can either be coded directly into the process, or it can be programmed using a supervisor bridge. Refer to the section Supervisor Bridge in this chapter.

Refer also to the Reference Manual, the module supervisor in stdlib.

3.2 Restart of Processes in the Supervision Tree

If a one_for_all supervisor detects that one of its children has died, then the supervisor will terminate and restart all the processes it is supervising.

If a one_for_one supervisor detects that one its children has died, then the supervisor will terminate and restart only the process that died. However, if the supervisor itself terminates, it terminates all the children it supervises.

The restart behaviour will occur if the following conditions apply:

• the child process was a permanent process which dies for any reason; or
• the child process was a transient process which dies with abnormal exit; and
• the restart limit has not been exceeded.

A temporary process which dies will not cause any restart behaviour.

3.3 The Shutdown Protocol

The shutdown protocol is used when the supervisor decides to kill a child for any reason. This is achieved by evaluating shutdown(Pid, Shutdown), where Pid is the Pid of the child, and Shutdown is the termination timeout value supplied in the start specification. Pid must be linked to the supervisor.

The supervisor process sends an exit signal to the child process and waits for acknowledgment. The child process should handle the {'EXIT',ParentPid,shutdown} message and terminate with reason shutdown if it traps exit signals. If an acknowledgment is not received within the specified time, then the child is killed.

Alternatively, the child can be terminated by specifying Shutdown = infinity, or Shutdown = brutal_kill. The exact behaviour is as follows:

shutdown(Pid, brutal_kill) ->
    exit(Pid, kill);
shutdown(Pid, infinity) ->
    exit(Pid, shutdown),
    receive
        {'EXIT', Pid, shutdown} -> true
    end;
shutdown(Pid, Time) ->
    exit(Pid, shutdown),
    receive
        {'EXIT', Pid, shutdown} ->
            true
    after Time ->
        exit(Pid, kill)
    end.

When a supervisor has to terminate several children, it terminates them synchronously in the reverse order of creation.

3.4 The Restart Frequency Limit Mechanism

The supervisors have a built-in mechanism to limit the number of restarts which can occur in a given time interval. This is determined by the values of the two parameters MaxR and MaxT. If more than MaxR number of restarts occur in the last MaxT seconds, then the supervisor shuts down all the children it supervises and dies.

If a supervisor terminates, then the next higher level supervisor takes some action. It either restarts the lower level supervisor or dies.

The intention of the restart mechanism is to prevent a situation where a process repeatedly dies for the same reason, only to be restarted again.

After the pre-determined number of restarts, the supervisor itself will die and the next higher level supervisor tries to correct the error by applying its restart strategy, and so on.

If a particular level in the system is incapable of correcting a given error, then it will eventually give up trying and pass the responsibility higher up in the supervision tree.

At some level in this process we hope that the error will finally be corrected.

3.4.1 Interpretation of the Restart Frequency

Assume that a number of restarts are performed at times:

T(0), T(1), T(2), ...

We define the quantity Acc(N) as follows:

Acc(N) = 1 + Acc(N-1) * (1 - Dt(N)/MaxT) when Dt(N) <= MaxR
Acc(N) = 1                                    Otherwise

In the above expression, Dt(N) = T(N) - T(N-1).

Acc represents an accumulator which increments by one every time a new restart is performed. The value of the accumulator decays linearly to zero in the time period MaxT, which is the measurement period.

If Acc is greater than the threshold value MaxR, then the supervising process dies.

3.5 Dynamic Processes in the Supervision Tree

In addition to the static supervision tree, we can also add dynamic children. A new child is added to an existing supervision tree with the following call:

supervisor:start_child(SupName, StartSpec) -> Result

SupName is the name,or the process identity, of the supervisor. StartSpec is a 6-tuple which contains the start specification of the child process, including the name of the child.

Children which are added using start_child/2 behave in the same manner as children defined in the start specification of the supervisor, with the following important exception:

If a supervisor dies and is re-created, then all children which were dynamically added to the supervisor will be lost.

Any child, static or dynamic, can be stopped with the following call, which also stops the child in accordance with the shutdown specification for the child.

supervisor:terminate_child(Sup, ChildName)

A stopped child can be restarted with the following call:

supervisor:restart_child(Sup, ChildName)

A stopped child is permanently deleted with the following call, and cannot be restarted again:

supervisor:delete_child(Sup, ChildName)

As with start_child/2, any effects of dynamically adding or deleting children is lost if the supervisor itself restarts. In this case, the original start specification is used to restart the children.

3.6 The Supervisor Bridge

It can sometimes be useful to connect a process or a subsystem to the supervision tree, although it has not been designed with the supervision principles in mind. This can be accomplished by using an instance of the supervisor_bridge behaviour. A supervisor bridge is a process which sits in between a supervisor and the subsystem. It behaves like a real supervisor to its supervisor, but has a different interface than a real supervisor to its subsystem.

Supervision Bridge

The subsystem must choose one main process to be responsible for the subsystem, which the supervisor bridge will monitor. If this process terminates, the supervisor bridge will assume that the entire subsystem has terminated.

The supervisor_bridge behaviour must export two functions:

• init/1, which starts the subsystem and returns {ok, Pid, State}. Pid is the Pid of the main subsystem process, and State is any term.
• terminate/2, which is responsible for terminating the entire subsystem.

Suppose that we want to connect the vshlr server to a supervision tree. This server is described in the next section which is titled Servers. If we use vshlr_2 version of the server, we can connect it directly to a supervisor as it is written, since it uses the gen_server behaviour. If we use vshlr_1 version instead, we must write a supervisor bridge to connect the server to a supervisor.

-module(vshlr_sup).
-copyright('Copyright (c) 1991-97 Ericsson Telecom AB').
-vsn('$Revision: /main/release/2 $').

-behaviour(supervisor_bridge).

-export([start_link/0]).
-export([init/1, terminate/2]).

%% Creates a supervisor bridge for vshlr_1
start_link() ->
    supervisor_bridge:start_link({local, vshlr_sup}, vshlr_sup, []).

%% This function is supposed to start the vshlr server
init([]) ->
    vshlr_1:start(),
    case whereis(vshlr_1) of
        Pid when pid(Pid) -> {ok, Pid, []};
        undefined -> {error, start_error}
    end.

%% This function is supposed to stop the vshlr server
terminate(_Reason, []) ->
    vshlr_1:stop().

A supervisor child specification for this supervisor bridge looks as follows:

{vshlr_sup, {vshlr_sup, start_link, []}, permanent, 2000, worker, []}

3.7 C Code and Supervision

Parts of a system are likely to be written in C. This may be for perfomance reasons, or as a result of sourcing. It is possible to make these programs to behave as supervised Erlang processes by wrapping the C programs with an Erlang stub, and this Erlang stub can be included as a worker in a supervision tree. The stub also relays supervision messages to the C program. The C program reacts to them in the form of "hooks", instead of messages.

The supervision features is included in the interface generator module. Using this module, C programs with their Erlang interfaces can benefit from being supervised.

Refer to the chapter titled Interface Libraries, the section The C Interface Generator for a detailed explanation of C code supervision.