5 Test Structure
5.1 Test structure
A test is performed by running one or more test suites. A test suite consists of test cases (as well as configuration functions and info functions). Test cases may be grouped in so called test case groups. A test suite is an Erlang module and test cases are implemented as Erlang functions. Test suites are stored in test directories.
5.2 Skipping test cases
It is possible to skip certain test cases, for example if you know beforehand that a specific test case fails. This might be functionality which isn't yet implemented, a bug that is known but not yet fixed or some functionality which doesn't work or isn't applicable on a specific platform.
There are several different ways to state that one or more test cases should be skipped:
- Using skip_suites and skip_cases terms in test specifications.
- Returning {skip,Reason} from the init_per_testcase/2 or init_per_suite/1 functions.
- Returning {skip,Reason} from the execution clause of the test case.
The latter of course means that the execution clause is actually called, so the author must make sure that the test case does not run.
When a test case is skipped, it will be noted as SKIPPED in the HTML log.
5.3 Definition of terms
- Auto skipped test case
- When a configuration function fails (i.e. terminates unexpectedly), the test cases that depend on the configuration function will be skipped automatically by Common Test. The status of the test cases is then "auto skipped". Test cases are also auto skipped by Common Test if required configuration data is not available at runtime.
- Configuration function
- A function in a test suite that is meant to be used for setting up, cleaning up, and/or verifying the state and environment on the SUT (System Under Test) and/or the Common Test host node, so that a test case (or a set of test cases) can execute correctly.
- Configuration file
- A file that contains data related to a test and/or an SUT (System Under Test), e.g. protocol server addresses, client login details, hardware interface addresses, etc - any data that should be handled as variable in the suite and not be hardcoded.
- Configuration variable
- A name (an Erlang atom) associated with a data value read from a configuration file.
- data_dir
- Data directory for a test suite. This directory contains any files used by the test suite, e.g. additional Erlang modules, binaries or data files.
- Info function
- A function in a test suite that returns a list of properties (read by the Common Test server) that describes the conditions for executing the test cases in the suite.
- Major log file
- An overview and summary log file for one or more test suites.
- Minor log file
- A log file for one particular test case. Also called the test case log file.
- priv_dir
- Private directory for a test suite. This directory should be used when the test suite needs to write to files.
- ct_run
- The name of an executable program that may be used as an interface for specifying and running tests with Common Test.
- Test case
- A single test included in a test suite. A test case is implemented as a function in a test suite module.
- Test case group
- A set of test cases that share configuration functions and execution properties. The execution properties specify whether the test cases in the group should be executed in random order, in parallel, in sequence, and if the execution of the group should be repeated. Test case groups may also be nested (i.e. a group may, besides test cases, contain sub-groups).
- Test suite
- An erlang module containing a collection of test cases for a specific functional area.
- Test directory
- A directory that contains one or more test suite modules, i.e. a group of test suites.
- The Config argument
- A list of key-value tuples (i.e. a property list) containing runtime configuration data passed from the configuration functions to the test cases.
- User skipped test case
- This is the status of a test case that has been explicitly skipped in any of the ways described in the "Skipping test cases" section above.