View Source Common Test Hooks
General
The Common Test Hook (CTH) framework allows extensions of the default behavior
of Common Test
using hooks before and after all test suite calls. CTHs allow
advanced Common Test
users to abstract out behavior that is common to multiple
test suites without littering all test suites with library calls. This can be
used for logging, starting, and monitoring external systems, building C files
needed by the tests, and so on.
In brief, CTH allows you to do the following:
- Manipulate the runtime configuration before each suite configuration call.
- Manipulate the return of all suite configuration calls, and in extension, the result of the tests themselves.
The following sections describe how to use CTHs, when they are run, and how to manipulate the test results in a CTH.
Warning
When executing within a CTH, all timetraps are shut off. So if your CTH never returns, the entire test run is stalled.
Installing a CTH
A CTH can be installed in multiple ways in your test run. You can do it for all tests in a run, for specific test suites, and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run, there are three ways to accomplish that, as follows:
- Add
-ct_hooks
as an argument to ct_run. To add multiple CTHs using this method, append them to each other using the keywordand
, that is,ct_run -ct_hooks cth1 [{debug,true}] and cth2 ...
. - Add tag
ct_hooks
to your Test Specification. - Add tag
ct_hooks
to your call toct:run_test/1
.
CTHs can also be added within a test suite. This is done by returning
{ct_hooks,[CTH]}
in the configuration list from
suite/0,
init_per_suite/1, or
init_per_group/2.
In this case, CTH
can either be only the module name of the CTH or a tuple
with the module name and the initial arguments, and optionally the hook priority
of the CTH. For example, one of the following:
{ct_hooks,[my_cth_module]}
{ct_hooks,[{my_cth_module,[{debug,true}]}]}
{ct_hooks,[{my_cth_module,[{debug,true}],500}]}
Note that regardless of how you install a CTH, its BEAM file must be available
in the code path when Common Test runs. ct_run
accepts the -pa
command line
option.
Overriding CTHs
By default, each installation of a CTH causes a new instance of it to be
activated. This can cause problems if you want to override CTHs in test
specifications while still having them in the suite information function. The
id/1 callback exists to address this problem. By returning
the same id
in both places, Common Test
knows that this CTH is already
installed and does not try to install it again.
CTH Execution Order
By default, each installed CTH is executed in the order in which they are
installed for init calls, and then reversed for end calls. This order can be
referred to as test-centric, as the order is reversed after a testcase is
executed and corresponds to the default value (test
) of ct_hooks_order
option.
The installation-based order is not always desired, so Common Test
allows the
user to specify a priority for each hook. The priority can be specified in the
CTH function init/2 or when installing the hook. The
priority specified at installation overrides the priority returned by the CTH.
In some cases, the reversed order for all end calls is not desired, and instead,
the user might prefer the reversed order for post hook calls. Such behavior can
be enabled with ct_hooks_order
option with config
value. When this option is
enabled, the execution order is configuration-centric, as the reversed order
happens after each configuration function and not in relation to testcase.
Note that the ct_hooks_order
option is considered as a global framework
setting. In case when option is configured multiple times framework with process
only the first value.
The ct_hooks_order
option can be set as: ct_run
argument, in test
specification or suite/0 return value.
CTH Scope
Once the CTH is installed into a certain test run it remains there until its scope is expired. The scope of a CTH depends on when it is installed, see the following table. Function init/2 is called at the beginning of the scope and function terminate/1 is called when the scope ends.
CTH installed in | CTH scope begins before | CTH scope ends after |
---|---|---|
ct_run | the first test suite is to be run | the last test suite has been run |
ct:run_test | the first test suite is run | the last test suite has been run |
Test Specification | the first test suite is run | the last test suite has been run |
suite/0 | pre_init_per_suite/3 is called | post_end_per_suite/4 has been called for that test suite |
init_per_suite/1 | post_init_per_suite/4 is called | post_end_per_suite/4 has been called for that test suite |
init_per_group/2 | post_init_per_group/5 is called | post_end_per_group/5 has been called for that group |
Table: Scope of a CTH
CTH Processes and Tables
CTHs are run with the same process scoping as normal test suites, that is, a
different process executes the init_per_suite
hooks then the init_per_group
or per_testcase
hooks. So if you want to spawn a process in the CTH, you
cannot link with the CTH process, as it exits after the post hook ends. Also, if
you for some reason need an ETS table with your CTH, you must spawn a process
that handles it.
External Configuration Data and Logging
Configuration data values in the CTH can be read by calling
ct:get_config/1,2,3
(as explained in section
Requiring and Reading Configuration Data).
The configuration variables in question must, as always, first have been
required by a suite-, group-, or test case information function, or by function
ct:require/1/2
. The latter can also be used in CT hook
functions.
The CT hook functions can call any logging function in the ct
interface to
print information to the log files, or to add comments in the suite overview
page.
Manipulating Tests
Through CTHs the results of tests and configuration functions can be manipulated. The main purpose to do this with CTHs is to allow common patterns to be abstracted out from test suites and applied to multiple test suites without duplicating any code. All the callback functions for a CTH follow a common interface described hereafter.
Common Test
always calls all available hook functions, even pre- and post
hooks for configuration functions that are not implemented in the suite. For
example, pre_init_per_suite(x_SUITE, ...)
and
post_init_per_suite(x_SUITE, ...)
are called for test suite x_SUITE
, even if
it does not export init_per_suite/1
. With this feature hooks can be used as
configuration fallbacks, and all configuration functions can be replaced with
hook functions.
Pre Hooks
In a CTH, the behavior can be hooked in before the following functions:
This is done in the CTH functions called pre_<name of function>
. These
functions take the arguments SuiteName
, Name
(group or test case name, if
applicable), Config
, and CTHState
. The return value of the CTH function is
always a combination of a result for the suite/group/test and an updated
CTHState
.
To let the test suite continue on executing, return the configuration list that you want the test to use as the result.
All pre hooks, except pre_end_per_testcase/4
, can skip or fail the test by
returning a tuple with skip
or fail
, and a reason as the result.
Example:
pre_init_per_suite(SuiteName, Config, CTHState) ->
case db:connect() of
{error,_Reason} ->
{{fail, "Could not connect to DB"}, CTHState};
{ok, Handle} ->
{[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
end.
Note
If you use multiple CTHs, the first part of the return tuple is used as input for the next CTH. So in the previous example the next CTH can get
{fail,Reason}
as the second parameter. If you have many CTHs interacting, do not let each CTH returnfail
orskip
. Instead, return that an action is to be taken through theConfig
list and implement a CTH that, at the end, takes the correct action.
Post Hooks
In a CTH, behavior can be hooked in after the following functions:
This is done in the CTH functions called post_<name of function>
. These
functions take the arguments SuiteName
, Name
(group or test case name, if
applicable), Config
, Return
, and CTHState
. Config
in this case is the
same Config
as the testcase is called with. Return
is the value returned by
the testcase. If the testcase fails by crashing, Return
is
{'EXIT',{{Error,Reason},Stacktrace}}
.
The return value of the CTH function is always a combination of a result for the
suite/group/test and an updated CTHState
. If you do not want the callback to
affect the outcome of the test, return the Return
data as it is given to the
CTH. You can also modify the test result. By returning the Config
list with
element tc_status
removed, you can recover from a test failure. As in all the
pre hooks, it is also possible to fail/skip the test case in the post hook.
Example:
post_end_per_testcase(_Suite, _TC, Config, {'EXIT',{_,_}}, CTHState) ->
case db:check_consistency() of
true ->
%% DB is good, pass the test.
{proplists:delete(tc_status, Config), CTHState};
false ->
%% DB is not good, mark as skipped instead of failing
{{skip, "DB is inconsistent!"}, CTHState}
end;
post_end_per_testcase(_Suite, _TC, Config, Return, CTHState) ->
%% Do nothing if tc does not crash.
{Return, CTHState}.
Note
Do recover from a testcase failure using CTHs only a last resort. If used wrongly, it can be very difficult to determine which tests that pass or fail in a test run.
Skip and Fail Hooks
After any post hook has been executed for all installed CTHs, on_tc_fail or on_tc_skip is called if the testcase failed or was skipped, respectively. You cannot affect the outcome of the tests any further at this point.
Synchronizing External User Applications with Common Test
CTHs can be used to synchronize test runs with external user applications. The
init function can, for example, start and/or communicate with an application
that has the purpose of preparing the SUT for an upcoming test run, or
initialize a database for saving test data to during the test run. The terminate
function can similarly order such an application to reset the SUT after the test
run, and/or tell the application to finish active sessions and terminate. Any
system error- or progress reports generated during the init- or termination
stage are saved in the
Pre- and Post Test I/O Log. (This is
also true for any printouts made with ct:log/2
and ct:pal/2
).
To ensure that Common Test
does not start executing tests, or closes its log
files and shuts down, before the external application is ready for it,
Common Test
can be synchronized with the application. During startup and
shutdown, Common Test
can be suspended, simply by having a CTH evaluate a
receive
expression in the init- or terminate function. The macros
?CT_HOOK_INIT_PROCESS
(the process executing the hook init function) and
?CT_HOOK_TERMINATE_PROCESS
(the process executing the hook terminate function)
each specifies the name of the correct Common Test
process to send a message
to. This is done to return from the receive
. These macros are defined in
ct.hrl
.
Example CTH
The following CTH logs information about a test run into a format parseable by
file:consult/1
(in Kernel):
%%% Common Test Example Common Test Hook module.
%%%
%%% To use this hook, on the command line:
%%% ct_run -suite example_SUITE -pa . -ct_hooks example_cth
%%%
%%% Note `-pa .`: the hook beam file must be in the code path when installing.
-module(example_cth).
%% Mandatory Callbacks
-export([init/2]).
%% Optional Callbacks
-export([id/1]).
-export([pre_init_per_suite/3]).
-export([post_end_per_suite/4]).
-export([pre_init_per_testcase/4]).
-export([post_end_per_testcase/5]).
-export([on_tc_skip/4]).
-export([terminate/1]).
%% This hook state is threaded through all the callbacks.
-record(state, {filename, total, suite_total, ts, tcs, data, skipped}).
%% This example hook prints its results to a file, see terminate/1.
-record(test_run, {total, skipped, suites}).
%% Return a unique id for this CTH.
%% Using the filename means the hook can be used with different
%% log files to separate timing data within the same test run.
%% See Installing a CTH for more information.
id(Opts) ->
%% the path is relative to the test run directory
proplists:get_value(filename, Opts, "example_cth.log").
%% Always called before any other callback function. Use this to initiate
%% any common state.
init(Id, _Opts) ->
{ok, #state{filename = Id, total = 0, data = []}}.
%% Called before init_per_suite is called.
pre_init_per_suite(_Suite,Config,State) ->
{Config, State#state{suite_total = 0, tcs = []}}.
%% Called after end_per_suite.
post_end_per_suite(Suite,_Config,Return,State) ->
Data = {suites, Suite, State#state.suite_total,
lists:reverse(State#state.tcs)},
{Return, State#state{data = [Data | State#state.data],
total = State#state.total + State#state.suite_total}}.
%% Called before each init_per_testcase.
pre_init_per_testcase(_Suite,_TC,Config,State) ->
Now = erlang:monotonic_time(microsecond),
{Config, State#state{ts = Now, suite_total = State#state.suite_total + 1}}.
%% Called after each end_per_testcase.
post_end_per_testcase(Suite,TC,_Config,Return,State) ->
Now = erlang:monotonic_time(microsecond),
TCInfo = {testcase, Suite, TC, Return, Now - State#state.ts},
{Return, State#state{ts = undefined, tcs = [TCInfo | State#state.tcs]}}.
%% Called when a test case is skipped by either user action
%% or due to an init function failing.
on_tc_skip(_Suite, _TC, _Reason, State) ->
State#state{skipped = State#state.skipped + 1}.
%% Called when the scope of the CTH is done.
terminate(State) ->
%% use append to avoid data loss if the path is reused
{ok, File} = file:open(State#state.filename, [write, append]),
io:format(File, "~p.~n", [results(State)]),
file:close(File),
ok.
results(State) ->
#state{skipped = Skipped, data = Data, total = Total} = State,
#test_run{total = Total, skipped = Skipped, suites = lists:reverse(Data)}.
Built-In CTHs
Common Test
is delivered with some general-purpose CTHs that can be enabled by
the user to provide generic testing functionality. Some of these CTHs are
enabled by default when common_test
is started to run. They can be disabled by
setting enable_builtin_hooks
to false
on the command line or in the test
specification. The following two CTHs are delivered with Common Test
:
cth_log_redirect
- Built-inCaptures all log events that would normally be printed by the default logger handler, and prints them to the current test case log. If an event cannot be associated with a test case, it is printed in the
Common Test
framework log. This happens for test cases running in parallel and events occurring in-between test cases.The log events are handled using a Logger handler called cth_log_redirect. The formatting and level is copied from the current
default
handler when the cth is started. If you want to use another level either change thedefault
handler level before starting common_test, or use thelogger:set_handler_config/3
API.This hook supports the following options:
{mode, add}
- Addcth_log_redirect
to the default logging handler: Logs will be emitted to both standard output via the default handler, and into the Common Test HTML logs. This is the default behaviour.{mode, replace}
- Replace thedefault
logging handler withcth_log_redirect
instead of logging to both the default handler and this handler. This effectively silences any logger output which would normally be printed to standard output during test runs. To enable this mode, you can pass the following options toct_run
:-enable_builtin_hooks false -ct_hooks cth_log_redirect [{mode,replace}]
cth_surefire
- Not built-inCaptures all test results and outputs them as surefire XML into a file. The created file is by default called
junit_report.xml
. The file name can be changed by setting optionpath
for this hook, for example:-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]
If option
url_base
is set, an extra attribute namedurl
is added to eachtestsuite
andtestcase
XML element. The value is constructed fromurl_base
and a relative path to the test suite or test case log, respectively, for example:-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]
gives an URL attribute value similar to
"http://myserver.com/ct_run.ct@myhost.2012-12-12_11.19.39/ x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"
Surefire XML can, for example, be used by Jenkins to display test results.