[erlang-questions] Using -spec for callbacks when defining behaviours

[Kostis Sagonas]

Mikael Karlsson wrote:
> Hi,
> 
> I understand that -spec (and -type)  constructs may be used in include
> (.hrl) files and the module itself when specifying functions.
> 
> Has there been any discussion on beeing able to use them in behaviour
> defining modules to specify the "callback" modules in more detail?
> 
> When defining a behaviour today you "just" specify the function names and
> their arity with the behaviour_info/1 function.
> 
> I think it would be good to be able to also specify the types of the
> arguments and their return type for the module implementing the behaviour.
> This would make a behaviour defining module closer to a interface
> specification and also decrease the dependencies of include files.

Absolutely!

In fact, we have had similar thoughts to yours and have been working for 
about half a year now on extending Erlang to do exactly what you are 
hinting at in your mail.  The good news is that it's all done and 
finished (dialyzer already supports this) and for about a month now 
we've even written an EEP for it, but I've been so swamped that I've not 
found time to properly send it to those responsible for EEPs.

I am including it below -- comments welcome,

Kostis

PS. Can I also please ask the folks of the Erlang/OTP team to put this 
in the EEP homepage?

============================================================================

EEP: XXX
Title: Behaviour Specifications
Version: $Revision: 1 $
Last-Modified: $Date: Mon Mar 8 12:14:35 EET 2010$
Author: Stavros Aronis [aronisstav(at)gmail(dot)com], Kostis Sagonas 
[kostis(at)cs(dot)ntua(dot)gr
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 03-Feb-2010
Erlang-Version:
Post-History:


Abstract
=======

We describe a small, backwards compatible change for behaviours to 
specify the types which are expected of their callbacks.


Rationale
=======

In Erlang a behaviour is usually a complete design pattern implemented 
in a module. Users can use these design patterns easily, writing in some 
other module just the callback functions required to provide the 
behaviour's functionality. As all these callbacks are called somewhere 
within the behaviour's code, it is important to check that behaviour 
callbacks have the functionality which is expected from them. Currently, 
not only is this the responsibility of the programmer but this checking 
is not even semi-automatic as there is no formal way for the module 
defining the behaviour to specify what is expected from its callback 
module. Instead, the programmer must check manually if the callback 
functions accept the correct arguments and return the correct results, 
and are indeed in accordance with the behaviour's available 
documentation, which often only exists in paper form. What this EEP aims 
at is to propose infrastructure which will ease the automatization of 
checking that callback modules adhere to what's expected from them.

Given the wide usage of behaviours in current Erlang application 
development, we believe that providing a means to include such 
specification in a behaviour's implementation is useful for both the 
behaviour's implementor, who can make clear what the types of the 
callbacks should be and the behaviour's user who can employ tools that 
check automatically for errors in the callback module's implementation.


Specification
==========

Currently, all behaviours must define a behaviour_info/1 function. Among 
other pieces of information, a clause of this function enumerates (in 
the form of 2-tuples) the function names and arities of the required 
callbacks.

The following example is taken from the 'gen_server' behaviour:

behaviour_info(callbacks) ->
   [{init,1}, {handle_call,3}, {handle_cast,2}, {terminate, 2}, 
{code_change, 3}].

Often, though not always, the behaviour module also contains some 
additional information in the form of comments. Continuing the same 
example, the gen_server module includes the following comments:

%%% The user module should export:
%%%
%%% init(Args)
%%% ==> {ok, State}
%%% {ok, State, Timeout}
%%% ignore
%%% {stop, Reason}
%%%
%%% handle_call(Msg, {From, Tag}, State)
%%%
%%% ==> {reply, Reply, State}
%%% {reply, Reply, State, Timeout}
%%% {noreply, State}
%%% {noreply, State, Timeout}
%%% {stop, Reason, Reply, State}
%%% Reason = normal | shutdown | Term terminate(State) is called
%%%
%%% .... MORE COMMENTS FOR THE OTHER THREE CALLBACKS HERE .....

The problem with comments is that they are in free text form, often 
lacking some information as in the case above, and cannot be 
mechanically processed.

We propose a modification in the behaviour_info(callbacks) clause so 
that it also specifies the types which are expected from these callbacks.

The modification itself is very simple, adding a third element in the 
tuple list.

The third element is a string whose contents is a -spec in the already 
existing language of EEP8:

behaviour_info(callbacks) ->
   [{init, 1,
     "-spec init(Args) ->
            {'ok', State} |
            {'ok', State, timeout() | 'hibernate'} |
            {'stop', Reason} |
            'ignore'."},"},
    {handle_call, 3,
     "-spec handle_call(Request, From :: {pid(), Tag}, State) ->
              {'reply', Reply, NewState} |
              {'reply', Reply, NewState, timeout() | 'hibernate'} |
              {'noreply', NewState} |
              {'noreply', NewState, timeout() | 'hibernate'} |
              {'stop', Reason, Reply, NewState} |
              {'stop', Reason, NewState}."},
    {handle_cast, 2, "-spec ... SOME SPEC HERE ..."},
    ....].

A defect detection tool (e.g. dialyzer) can then use these specs as a 
reference to compare the inferred types of the callbacks.

Incidentally, the above example shows various interesting things:

1) Using the language of types and specs, one can provide information 
both for documentation purposes and for types as e.g. in From :: {pid(), 
Tag}

2) It's not necessary to specify every type as e.g. the Tag variable 
above, which is a convenient shorthand for Tag :: term()

3) Comments are often incomplete or can easily become obsolete as e.g. 
the 'hibernate' value is nowhere mentioned.

A final note: One may wonder why the specs are written as strings 
instead of as -specs. There are various reasons for this, though none of 
them is deep or cast in stone. First of all specs are not proper Erlang 
terms (they are file attributes), so they cannot be used as is in places 
where Erlang terms are expected. Since we wanted to reuse as much of the 
existing infrastructure of behaviours as possible, the natural place to 
put them was to extend the behaviour_info(callbacks) clause. Second, 
they specify a type obligation (i.e., a contract) for all the callback 
modules, not for some specific known module or the behaviour module that 
contains them (the behaviour module does not contain definitions for 
these functions). Because of this, they cannot be placed in the 
behaviour defining module since currently there is no way for the 
language of specs to express something of the form: for all callback 
modules M, -spec M:init(Args) -> ....
If we use a file attribute other than -spec (e.g. -callback_spec) these 
specs could be part of the module defining the behaviour instead of 
being part of the behaviour_info(callbacks) clause.


Implementation
==============

We have already implemented the above proposal and we can provide a 
fully working implementation.

Trivial changes were needed in the compiler to accept the new 
behaviour_info format when checking for the presence of all the required 
callbacks in a module that uses the behaviour, as well as minor changes 
in some other library modules. Using this extension of dialyzer in a 
significant corpus of Erlang code we have already detected many 
violations of the published behaviour documentation.


Backwards Compatibility
=======================

The extension is fully backwards compatible. The old format is still 
supported (the clause of the behaviour_info function can contain both 
pairs and triples).

The inclusion of the -spec string is optional and no warnings are 
emitted unless Dialyzer is asked explicitly to check the behaviour usage 
via a corresponding option.


Copyright
=========

This document has been placed in the public domain.

..

Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8