13 Common Test Hooks

13.1 General

The Common Test Hook (henceforth called CTH) framework allows extensions of the default behaviour of Common Test by means of hooks before and after all test suite calls. CTHs allow advanced Common Test users to abstract out behaviour which is common to multiple test suites without littering all test suites with library calls. Some example usages are: logging, starting and monitoring external systems, building C files needed by the tests and much more!

In brief, Common Test Hooks allows you to:

Manipulate the runtime config before each suite configuration call
Manipulate the return of all suite configuration calls and in extension the result of the test themselves.

The following sections describe how to use CTHs, when they are run and how to manipulate your test results in a CTH

Warning

When executing within a CTH all timetraps are shutoff. So if your CTH never returns, the entire test run will be stalled!

13.2 Installing a CTH

There are multiple ways to install a CTH in your test run. You can do it for all tests in a run, for specific test suites and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run there are three different ways to accomplish that.

Add -ct_hooks as an argument to ct_run. To add multiple CTHs using this method append them to each other using the keyword and, i.e. ct_run -ct_hooks cth1 [{debug,true}] and cth2 ....
Add the ct_hooks tag to your Test Specification
Add the ct_hooks tag to your call to ct:run_test/1

You can also add CTHs within a test suite. This is done by returning {ct_hooks,[CTH]} in the config list from suite/0, init_per_suite/1 or init_per_group/2. CTH in this case can be either only the module name of the CTH or a tuple with the module name and the initial arguments and optionally the hook priority of the CTH. Eg: {ct_hooks,[my_cth_module]} or {ct_hooks,[{my_cth_module,[{debug,true}]}]} or {ct_hooks,[{my_cth_module,[{debug,true}],500}]}

Overriding CTHs

By default each installation of a CTH will cause a new instance of it to be activated. This can cause problems if you want to be able to override CTHs in test specifications while still having them in the suite info function. The id/1 callback exists to address this problem. By returning the same id in both places, Common Test knows that this CTH has already been installed and will not try to install it again.

CTH Execution order

By default each CTH installed will be executed in the order which they are installed for init calls, and then reversed for end calls. This is not always wanted so common_test allows the user to specify a priority for each hook. The priority can either be specified in the CTH init/2 function or when installing the hook. The priority given at installation will override the priority returned by the CTH.

13.3 CTH Scope

Once the CTH is installed into a certain test run it will be there until its scope is expired. The scope of a CTH depends on when it is installed. The init/2 is called at the beginning of the scope and the terminate/1 function is called when the scope ends.

CTH Installed in	CTH scope begins before	CTH scope ends after
ct_run	the first test suite is to be run.	the last test suite has been run.
ct:run_test	the first test suite is to be run.	the last test suite has been run.
Test Specification	the first test suite is to be run.	the last test suite has been run.
suite/0	pre_init_per_suite/3 is called.	post_end_per_suite/4 has been called for that test suite.
init_per_suite/1	post_init_per_suite/4 is called.	post_end_per_suite/4 has been called for that test suite.
init_per_group/2	post_init_per_group/4 is called.	post_end_per_group/4 has been called for that group.

Table 13.1: Scope of a CTH

CTH Processes and Tables

CTHs are run with the same process scoping as normal test suites i.e. a different process will execute the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH you cannot link with the CTH process as it will exit after the post hook ends. Also if you for some reason need an ETS table with your CTH, you will have to spawn a process which handles it.

External configuration data and Logging

It's possible in the CTH to read configuration data values by calling ct:get_config/1/2/3 (as explained in the External configuration data chapter). The config variables in question must, as always, first have been required by means of a suite-, group-, or test case info function, or the ct:require/1/2 function. Note that the latter can also be used in CT hook functions.

The CT hook functions may call any of the logging functions available in the ct interface to print information to the log files, or to add comments in the suite overview page.

13.4 Manipulating tests

It is through CTHs possible to manipulate the results of tests and configuration functions. The main purpose of doing this with CTHs is to allow common patterns to be abstracted out from test test suites and applied to multiple test suites without duplicating any code. All of the callback functions for a CTH follow a common interface, this interface is described below.

Common Test will always call all available hook functions, even pre- and post hooks for configuration functions that are not implemented in the suite. For example, pre_init_per_suite(x_SUITE, ...) and post_init_per_suite(x_SUITE, ...) will be called for test suite x_SUITE, even if it doesn't export init_per_suite/1. This feature makes it possible to use hooks as configuration fallbacks, or even completely replace all configuration functions with hook functions.

Pre Hooks

It is possible in a CTH to hook in behaviour before init_per_suite, init_per_group, init_per_testcase, end_per_group and end_per_suite. This is done in the CTH functions called pre_<name of function>. All of these functions take the same three arguments: Name, Config and CTHState. The return value of the CTH function is always a combination of an result for the suite/group/test and an updated CTHState. If you want the test suite to continue on executing you should return the config list which you want the test to use as the result. If you for some reason want to skip/fail the test, return a tuple with skip or fail and a reason as the result. Example:

pre_init_per_suite(SuiteName, Config, CTHState) ->
  case db:connect() of
    {error,_Reason} ->
      {{fail, "Could not connect to DB"}, CTHState};
    {ok, Handle} ->
      {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
  end.

Note

If using multiple CTHs, the first part of the return tuple will be used as input for the next CTH. So in the case above the next CTH might get {fail,Reason} as the second parameter. If you have many CTHs which interact, it might be a good idea to not let each CTH return fail or skip. Instead return that an action should be taken through the Config list and implement a CTH which at the end takes the correct action.

Post Hooks

It is also possible in a CTH to hook in behaviour after init_per_suite, init_per_group, end_per_testcase, end_per_group and end_per_suite. This is done in the CTH functions called post_<name of function>. All of these function take the same four arguments: Name, Config, Return and CTHState. Config in this case is the same Config as the testcase is called with. Return is the value returned by the testcase. If the testcase failed by crashing, Return will be {'EXIT',{{Error,Reason},Stacktrace}}.

The return value of the CTH function is always a combination of an result for the suite/group/test and an updated CTHState. If you want the callback to not affect the outcome of the test you should return the Return data as it is given to the CTH. You can also modify the result of the test. By returning the Config list with the tc_status element removed you can recover from a test failure. As in all the pre hooks, it is also possible to fail/skip the test case in the post hook. Example:

post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) ->
  case db:check_consistency() of
    true ->
      %% DB is good, pass the test.
      {proplists:delete(tc_status, Config), CTHState};
    false ->
      %% DB is not good, mark as skipped instead of failing
      {{skip, "DB is inconsisten!"}, CTHState}
  end;
post_end_per_testcase(_TC, Config, Return, CTHState) ->
  %% Do nothing if tc does not crash.
  {Return, CTHState}.

Note

Recovering from a testcase failure using CTHs should only be done as a last resort. If used wrongly it could become very difficult to determine which tests pass or fail in a test run

Skip and Fail hooks

After any post hook has been executed for all installed CTHs, on_tc_fail or on_tc_skip might be called if the testcase failed or was skipped respectively. You cannot affect the outcome of the tests any further at this point.

13.5 Synchronizing external user applications with Common Test

CTHs can be used to synchronize test runs with external user applications. The init function may e.g. start and/or communicate with an application that has the purpose of preparing the SUT for an upcoming test run, or maybe initialize a database for saving test data to during the test run. The terminate function may similarly order such an application to reset the SUT after the test run, and/or tell the application to finish active sessions and terminate. Any system error- or progress reports generated during the init- or termination stage will be saved in the Pre- and post test I/O log. (This is also true for any printouts made with ct:log/2 and ct:pal/2).

In order to ensure that Common Test doesn't start executing tests, or closes its log files and shuts down, before the external application is ready for it, Common Test may be synchronized with the application. During startup and shutdown, Common Test can be suspended, simply by having a CTH evaluate a receive expression in the init- or terminate function. The macros ?CT_HOOK_INIT_PROCESS (the process executing the hook init function) and ?CT_HOOK_TERMINATE_PROCESS (the process executing the hook terminate function), each specifies the name of the correct Common Test process to send a message to in order to return from the receive. These macros are defined in ct.hrl.

13.6 Example CTH

The CTH below will log information about a test run into a format parseable by file:consult/1.

%%% @doc Common Test Example Common Test Hook module.
-module(example_cth).

%% Callbacks
-export([id/1]).
-export([init/2]).

-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
-export([pre_end_per_suite/3]).
-export([post_end_per_suite/4]).

-export([pre_init_per_group/3]).
-export([post_init_per_group/4]).
-export([pre_end_per_group/3]).
-export([post_end_per_group/4]).

-export([pre_init_per_testcase/3]).
-export([post_end_per_testcase/4]).

-export([on_tc_fail/3]).
-export([on_tc_skip/3]).

-export([terminate/1]).

-record(state, { file_handle, total, suite_total, ts, tcs, data }).

%% @doc Return a unique id for this CTH.
id(Opts) ->
  proplists:get_value(filename, Opts, "/tmp/file.log").

%% @doc Always called before any other callback function. Use this to initiate
%% any common state. 
init(Id, Opts) ->
    {ok,D} = file:open(Id,[write]),
    {ok, #state{ file_handle = D, total = 0, data = [] }}.

%% @doc Called before init_per_suite is called. 
pre_init_per_suite(Suite,Config,State) ->
    {Config, State#state{ suite_total = 0, tcs = [] }}.

%% @doc Called after init_per_suite.
post_init_per_suite(Suite,Config,Return,State) ->
    {Return, State}.

%% @doc Called before end_per_suite. 
pre_end_per_suite(Suite,Config,State) ->
    {Config, State}.

%% @doc Called after end_per_suite. 
post_end_per_suite(Suite,Config,Return,State) ->
    Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
    {Return, State#state{ data = [Data | State#state.data] ,
                          total = State#state.total + State#state.suite_total } }.

%% @doc Called before each init_per_group.
pre_init_per_group(Group,Config,State) ->
    {Config, State}.

%% @doc Called after each init_per_group.
post_init_per_group(Group,Config,Return,State) ->
    {Return, State}.

%% @doc Called after each end_per_group. 
pre_end_per_group(Group,Config,State) ->
    {Config, State}.

%% @doc Called after each end_per_group. 
post_end_per_group(Group,Config,Return,State) ->
    {Return, State}.

%% @doc Called before each test case.
pre_init_per_testcase(TC,Config,State) ->
    {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.

%% @doc Called after each test case.
post_end_per_testcase(TC,Config,Return,State) ->
    TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
    {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.

%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
on_tc_fail(TC, Reason, State) ->
    State.

%% @doc Called when a test case is skipped by either user action
%% or due to an init function failing.  
on_tc_skip(TC, Reason, State) ->
    State.

%% @doc Called when the scope of the CTH is done
terminate(State) ->
    io:format(State#state.file_handle, "~p.~n",
               [{test_run, State#state.total, State#state.data}]),
    file:close(State#state.file_handle),
    ok.

13.7 Built-in CTHs

Common Test is delivered with a couple of general purpose CTHs that can be enabled by the user to provide some generic testing functionality. Some of these are enabled by default when starting running common_test, they can be disabled by setting enable_builtin_hooks to false on the command line or in the test specification. In the table below there is a list of all current CTHs which are delivered with Common Test.

CTH Name

Is Built-in

Description

cth_log_redirect

yes

Captures all error_logger and SASL logging events and prints them to the current test case log. If an event can not be associated with a testcase it will be printed in the common test framework log. This will happen for testcases which are run in parallel and events which occur inbetween testcases. You can configure the level of SASL events report using the normal SASL mechanisms.

cth_surefire

Captures all test results and outputs them as surefire XML into a file. The file which is created is by default called junit_report.xml. The file name can be changed by setting the path option for this hook, e.g.

-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]

If the url_base option is set, an additional attribute named url will be added to each testsuite and testcase XML element. The value will be constructed from the url_base and a relative path to the test suite or test case log respectively, e.g.

-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]

will give a url attribute value similar to

"http://myserver.com/ct_run.ct@myhost.2012-12-12_11.19.39/
x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"

Surefire XML can for instance be used by Jenkins to display test results.