12 Common Test Hooks
12.1 General
This feature is in alpha release right now. This means that the interface may change in the future and that there may be bugs. We encourage you to use this feature, but be prepared that there might be bugs and that the interface might change inbetween releases.
The Common Test Hook (henceforth called CTH) framework allows extensions of the default behaviour of Common Test by means of hooks before and after all test suite calls. CTHs allow advanced Common Test users to abstract out behaviour which is common to multiple test suites without littering all test suites with library calls. Some example usages are: logging, starting and monitoring external systems, building C files needed by the tests and much more!
In brief, Common Test Hooks allows you to:
- Manipulate the runtime config before each suite configuration call
- Manipulate the return of all suite configuration calls and in extension the result of the test themselves.
The following sections describe how to use CTHs, when they are run and how to manipulate your test results in a CTH
When executing within a CTH all timetraps are shutoff. So if your CTH never returns, the entire test run will be stalled!
12.2 Installing a CTH
There are multiple ways to install a CTH in your test run. You can do it for all tests in a run, for specific test suites and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run there are three different ways to accomplish that.
- Add -ct_hooks as an argument to ct_run. To add multiple CTHs using this method append them to each other using the keyword and, i.e. ct_run -ct_hooks cth1 [{debug,true}] and cth2 ....
- Add the ct_hooks tag to your Test Specification
- Add the ct_hooks tag to your call to ct:run_test/1
You can also add CTHs within a test suite. This is done by returning {ct_hooks,[CTH]} in the config list from suite/0, init_per_suite/1 or init_per_group/2. CTH in this case can be either only the module name of the CTH or a tuple with the module name and the initial arguments to the CTH. Eg: {ct_hooks,[my_cth_module]} or {ct_hooks,[{my_cth_module,[{debug,true}]}]}
Overriding CTHs
By default each installation of a CTH will cause a new instance of it to be activated. This can cause problems if you want to be able to override CTHs in test specifications while still having them in the suite info function. The id/1 callback exists to address this problem. By returning the same id in both places, Common Test knows that this CTH has already been installed and will not try to install it again.
12.3 CTH Scope
Once the CTH is installed into a certain test run it will be there until its scope is expired. The scope of a CTH depends on when it is installed. The init/2 is called at the beginning of the scope and the terminate/1 function is called when the scope ends.
| CTH Installed in | CTH scope begins before | CTH scope ends after |
| ct_run | the first test suite is to be run. | the last test suite has been run. |
| ct:run_test | the first test suite is to be run. | the last test suite has been run. |
| Test Specification | the first test suite is to be run. | the last test suite has been run. |
| suite/0 | pre_init_per_suite/3 is called. | post_end_per_suite/4 has been called for that test suite. |
| init_per_suite/1 | post_init_per_suite/4 is called. | post_end_per_suite/4 has been called for that test suite. |
| init_per_group/2 | post_init_per_group/4 is called. | post_end_per_group/4 has been called for that group. |
CTH Processes and Tables
CTHs are run with the same process scoping as normal test suites i.e. a different process will execute the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH you cannot link with the CTH process as it will exit after the post hook ends. Also if you for some reason need an ETS table with your CTH, you will have to spawn a process which handles it.
12.4 Manipulating tests
It is through CTHs possible to manipulate the results of tests and configuration functions. The main purpose of doing this with CTHs is to allow common patterns to be abstracted out from test test suites and applied to multiple test suites without duplicating any code. All of the callback functions for a CTH follow a common interface, this interface is described below.
It is only possible to hook into test function which exists in the test suite. So in order for a CTH to hook in before init_per_suite, the init_per_suite function must exist in the test suite.
Pre Hooks
It is possible in a CTH to hook in behaviour before init_per_suite, init_per_group, init_per_testcase, end_per_group and end_per_suite. This is done in the CTH functions called pre_<name of function>. All of these functions take the same three arguments: Name, Config and CTHState. The return value of the CTH function is always a combination of an result for the suite/group/test and an updated CTHState. If you want the test suite to continue on executing you should return the config list which you want the test to use as the result. If you for some reason want to skip/fail the test, return a tuple with skip or fail and a reason as the result. Example:
pre_init_per_suite(SuiteName, Config, CTHState) ->
case db:connect() of
{error,_Reason} ->
{{fail, "Could not connect to DB"}, CTHState};
{ok, Handle} ->
{[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
end.Post Hooks
It is also possible in a CTH to hook in behaviour after init_per_suite, init_per_group, end_per_testcase, end_per_group and end_per_suite. This is done in the CTH functions called post_<name of function>. All of these function take the same four arguments: Name, Config, Return and CTHState. Config in this case is the same Config as the testcase is called with. Return is the value returned by the testcase. If the testcase failed by crashing, Return will be {'EXIT',{{Error,Reason},Stacktrace}}.
The return value of the CTH function is always a combination of an result for the suite/group/test and an updated CTHState. If you want the callback to not affect the outcome of the test you should return the Return data as it is given to the CTH. You can also modify the result of the test. By returning the Config list with the tc_status element removed you can recover from a test failure. As in all the pre hooks, it is also possible to fail/skip the test case in the post hook. Example:
post_end_per_testcase(_TC, Config, {'EXIT',{_,_}}, CTHState) ->
case db:check_consistency() of
true ->
%% DB is good, pass the test.
{proplists:delete(tc_status, Config), CTHState};
false ->
%% DB is not good, mark as skipped instead of failing
{{skip, "DB is inconsisten!"}, CTHState}
end;
post_end_per_testcase(_TC, Config, Return, CTHState) ->
%% Do nothing if tc does not crash.
{Return, CTHState}.Recovering from a testcase failure using CTHs should only be done as a last resort. If used wrongly it could become very difficult to determine which tests pass or fail in a test run
Skip and Fail hooks
After any post hook has been executed for all installed CTHs, on_tc_fail or on_tc_skip might be called if the testcase failed or was skipped respectively. You cannot affect the outcome of the tests any further at this point.
12.5 Example CTH
The CTH below will log information about a test run into a format parseable by file:consult/1.
%%% @doc Common Test Example Common Test Hook module.
-module(example_cth).
%% Callbacks
-export([id/1]).
-export([init/2]).
-export([pre_init_per_suite/3]).
-export([post_init_per_suite/4]).
-export([pre_end_per_suite/3]).
-export([post_end_per_suite/4]).
-export([pre_init_per_group/3]).
-export([post_init_per_group/4]).
-export([pre_end_per_group/3]).
-export([post_end_per_group/4]).
-export([pre_init_per_testcase/3]).
-export([post_end_per_testcase/4]).
-export([on_tc_fail/3]).
-export([on_tc_skip/3]).
-export([terminate/1]).
-record(state, { file_handle, total, suite_total, ts, tcs, data }).
%% @doc Return a unique id for this CTH.
id(Opts) ->
proplists:get_value(filename, Opts, "/tmp/file.log").
%% @doc Always called before any other callback function. Use this to initiate
%% any common state.
init(Id, Opts) ->
{ok,D} = file:open(Id,[write]),
#state{ file_handle = D, total = 0, data = [] }.
%% @doc Called before init_per_suite is called.
pre_init_per_suite(Suite,Config,State) ->
{Config, State#state{ suite_total = 0, tcs = [] }}.
%% @doc Called after init_per_suite.
post_init_per_suite(Suite,Config,Return,State) ->
{Return, State}.
%% @doc Called before end_per_suite.
pre_end_per_suite(Suite,Config,State) ->
{Config, State}.
%% @doc Called after end_per_suite.
post_end_per_suite(Suite,Config,Return,State) ->
Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
{Return, State#state{ data = [Data | State#state.data] ,
total = State#state.total + State#state.suite_total } }.
%% @doc Called before each init_per_group.
pre_init_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each init_per_group.
post_init_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called after each end_per_group.
pre_end_per_group(Group,Config,State) ->
{Config, State}.
%% @doc Called after each end_per_group.
post_end_per_group(Group,Config,Return,State) ->
{Return, State}.
%% @doc Called before each test case.
pre_init_per_testcase(TC,Config,State) ->
{Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.
%% @doc Called after each test case.
post_end_per_testcase(TC,Config,Return,State) ->
TCInfo = {testcase, TC, Return, timer:now_diff(now(), State#state.ts)},
{Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.
%% @doc Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
%% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
on_tc_fail(TC, Reason, State) ->
State.
%% @doc Called when a test case is skipped by either user action
%% or due to an init function failing.
on_tc_skip(TC, Reason, State) ->
State.
%% @doc Called when the scope of the CTH is done
terminate(State) ->
io:format(State#state.file_handle, "~p.~n",
[{test_run, State#state.total, State#state.data}]),
file:close(State#state.file_handle),
ok.