[erlang-questions] Using -spec for callbacks when defining behaviours

[Mikael Karlsson]

2010/3/17 Kostis Sagonas <kostis@REDACTED>

> Mikael Karlsson wrote:
>
>> Hi,
>>
>> I understand that -spec (and -type)  constructs may be used in include
>> (.hrl) files and the module itself when specifying functions.
>>
>> Has there been any discussion on beeing able to use them in behaviour
>> defining modules to specify the "callback" modules in more detail?
>>
>> When defining a behaviour today you "just" specify the function names and
>> their arity with the behaviour_info/1 function.
>>
>> I think it would be good to be able to also specify the types of the
>> arguments and their return type for the module implementing the behaviour.
>> This would make a behaviour defining module closer to a interface
>> specification and also decrease the dependencies of include files.
>>
>
> Absolutely!
>
> In fact, we have had similar thoughts to yours and have been working for
> about half a year now on extending Erlang to do exactly what you are hinting
> at in your mail.  The good news is that it's all done and finished (dialyzer
> already supports this) and for about a month now we've even written an EEP
> for it, but I've been so swamped that I've not found time to properly send
> it to those responsible for EEPs.
>
> I am including it below -- comments welcome,
>
> Kostis
>
> PS. Can I also please ask the folks of the Erlang/OTP team to put this in
> the EEP homepage?
>
>
> ============================================================================
>
> EEP: XXX
> Title: Behaviour Specifications
> Version: $Revision: 1 $
> Last-Modified: $Date: Mon Mar 8 12:14:35 EET 2010$
> Author: Stavros Aronis [aronisstav(at)gmail(dot)com], Kostis Sagonas
> [kostis(at)cs(dot)ntua(dot)gr
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 03-Feb-2010
> Erlang-Version:
> Post-History:
>
>
> Abstract
> =======
>
> We describe a small, backwards compatible change for behaviours to specify
> the types which are expected of their callbacks.
>
>
> Rationale
> =======
>
> In Erlang a behaviour is usually a complete design pattern implemented in a
> module. Users can use these design patterns easily, writing in some other
> module just the callback functions required to provide the behaviour's
> functionality. As all these callbacks are called somewhere within the
> behaviour's code, it is important to check that behaviour callbacks have the
> functionality which is expected from them. Currently, not only is this the
> responsibility of the programmer but this checking is not even
> semi-automatic as there is no formal way for the module defining the
> behaviour to specify what is expected from its callback module. Instead, the
> programmer must check manually if the callback functions accept the correct
> arguments and return the correct results, and are indeed in accordance with
> the behaviour's available documentation, which often only exists in paper
> form. What this EEP aims at is to propose infrastructure which will ease the
> automatization of checking that callback modules adhere to what's expected
> from them.
>
> Given the wide usage of behaviours in current Erlang application
> development, we believe that providing a means to include such specification
> in a behaviour's implementation is useful for both the behaviour's
> implementor, who can make clear what the types of the callbacks should be
> and the behaviour's user who can employ tools that check automatically for
> errors in the callback module's implementation.
>
>
> Specification
> ==========
>
> Currently, all behaviours must define a behaviour_info/1 function. Among
> other pieces of information, a clause of this function enumerates (in the
> form of 2-tuples) the function names and arities of the required callbacks.
>
> The following example is taken from the 'gen_server' behaviour:
>
> behaviour_info(callbacks) ->
>  [{init,1}, {handle_call,3}, {handle_cast,2}, {terminate, 2}, {code_change,
> 3}].
>
> Often, though not always, the behaviour module also contains some
> additional information in the form of comments. Continuing the same example,
> the gen_server module includes the following comments:
>
> %%% The user module should export:
> %%%
> %%% init(Args)
> %%% ==> {ok, State}
> %%% {ok, State, Timeout}
> %%% ignore
> %%% {stop, Reason}
> %%%
> %%% handle_call(Msg, {From, Tag}, State)
> %%%
> %%% ==> {reply, Reply, State}
> %%% {reply, Reply, State, Timeout}
> %%% {noreply, State}
> %%% {noreply, State, Timeout}
> %%% {stop, Reason, Reply, State}
> %%% Reason = normal | shutdown | Term terminate(State) is called
> %%%
> %%% .... MORE COMMENTS FOR THE OTHER THREE CALLBACKS HERE .....
>
> The problem with comments is that they are in free text form, often lacking
> some information as in the case above, and cannot be mechanically processed.
>
> We propose a modification in the behaviour_info(callbacks) clause so that
> it also specifies the types which are expected from these callbacks.
>
> The modification itself is very simple, adding a third element in the tuple
> list.
>
> The third element is a string whose contents is a -spec in the already
> existing language of EEP8:
>
> behaviour_info(callbacks) ->
>  [{init, 1,
>    "-spec init(Args) ->
>           {'ok', State} |
>           {'ok', State, timeout() | 'hibernate'} |
>           {'stop', Reason} |
>           'ignore'."},"},
>   {handle_call, 3,
>    "-spec handle_call(Request, From :: {pid(), Tag}, State) ->
>             {'reply', Reply, NewState} |
>             {'reply', Reply, NewState, timeout() | 'hibernate'} |
>             {'noreply', NewState} |
>             {'noreply', NewState, timeout() | 'hibernate'} |
>             {'stop', Reason, Reply, NewState} |
>             {'stop', Reason, NewState}."},
>   {handle_cast, 2, "-spec ... SOME SPEC HERE ..."},
>   ....].
>
> A defect detection tool (e.g. dialyzer) can then use these specs as a
> reference to compare the inferred types of the callbacks.
>
> Incidentally, the above example shows various interesting things:
>
> 1) Using the language of types and specs, one can provide information both
> for documentation purposes and for types as e.g. in From :: {pid(), Tag}
>
> 2) It's not necessary to specify every type as e.g. the Tag variable above,
> which is a convenient shorthand for Tag :: term()
>
> 3) Comments are often incomplete or can easily become obsolete as e.g. the
> 'hibernate' value is nowhere mentioned.
>
> A final note: One may wonder why the specs are written as strings instead
> of as -specs. There are various reasons for this, though none of them is
> deep or cast in stone. First of all specs are not proper Erlang terms (they
> are file attributes), so they cannot be used as is in places where Erlang
> terms are expected. Since we wanted to reuse as much of the existing
> infrastructure of behaviours as possible, the natural place to put them was
> to extend the behaviour_info(callbacks) clause. Second, they specify a type
> obligation (i.e., a contract) for all the callback modules, not for some
> specific known module or the behaviour module that contains them (the
> behaviour module does not contain definitions for these functions). Because
> of this, they cannot be placed in the behaviour defining module since
> currently there is no way for the language of specs to express something of
> the form: for all callback modules M, -spec M:init(Args) -> ....
> If we use a file attribute other than -spec (e.g. -callback_spec) these
> specs could be part of the module defining the behaviour instead of being
> part of the behaviour_info(callbacks) clause.
>
>
> Implementation
> ==============
>
> We have already implemented the above proposal and we can provide a fully
> working implementation.
>
> Trivial changes were needed in the compiler to accept the new
> behaviour_info format when checking for the presence of all the required
> callbacks in a module that uses the behaviour, as well as minor changes in
> some other library modules. Using this extension of dialyzer in a
> significant corpus of Erlang code we have already detected many violations
> of the published behaviour documentation.
>
>
> Backwards Compatibility
> =======================
>
> The extension is fully backwards compatible. The old format is still
> supported (the clause of the behaviour_info function can contain both pairs
> and triples).
>
> The inclusion of the -spec string is optional and no warnings are emitted
> unless Dialyzer is asked explicitly to check the behaviour usage via a
> corresponding option.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
> ..
>
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> coding: utf-8
>

Great news!

As I can see from the following discussion there can be several things to
say about the how to implement this, nevertheless it is already in the works
which is really nice.

I have one question:
It is not clear to me how to declare -type in the behaviour_info function.
If I have a recurring pattern in several of the -specs it is of course nice
to be able to do this.

You mention "contract" as a "type obligation" in the final note. I read in a
little more in a contract, that is that you also have runtime checks in a
contract (like UBF contract checks or Eiffel contracts).
Anyway I think that it will be quite straightforward to include (runtime)
contract checking as well in behaviours, at least if we will be allowed to
officially play with parameterized (abstract) modules in the future, meaning
that we will have "everything" in place :

-module(behave).
-export([behaviour_info/1, contract_check/1]).

behaviour_info(callbacks)  ->  [{set,2},{get,1}]. %% Should of course use
new "-specs.."

contract_check(M) ->  behave_contract:new(M).

------
-module(behave_contract,[M]).
-behaviour(behave).
-export([set/2, get/1]).

get(A) -> M:get(A).

set(Key, Value) ->
    M1 = M:set(Key, Value),
    Value = M1:get(Key), %% Contract check
    new(M1).
----
Im my implementation module:
-ifdef(debug).
M = behave:contract_check(my_behave_impl:new(Args)),
-else.
M = my_behave_impl:new(Args),
-endif.
----
/Mikael