1 Test Server Basics
1.1 Introduction
Testing when using the test server is done by running test suites. A test suite is a number of test cases, where each test case tests one or more things. The test case is the smallest unit that the test server deals with. One or more test cases are grouped together into one ordinary Erlang module, which is called a test suite. Several test suite modules can be grouped together in special test specification files representing whole application and/or system test "jobs".
The test suite Erlang module must follow a certain interface, which is specified by the test server. See the section on writing test suites for details about this.
Each test case is considered a success if it returns to the caller, no matter what the returned value is. An exception to this is the return value
{skip, Reason}
which indicates that the test case is skipped. A failure is specified as a crash, no matter what the crash reason is.As a test suite runs, all information (including output to stdout) is recorded in several different log files. A minimum of information is displayed to the user console. This only include start and stop information, plus a note for each failed test case.
The result from each test case is recorded in an HTML log file which is created for each test run. Every test case gets one row in a table presenting total time, whether the case was successful or not, if it was skipped, and possibly also a comment. The HTML file has links to each test case's logfile, wich may be viewed from e.g. Netscape or any other HTML capable browser.
The Test Server consists of three parts:
- The part that executes the test suites on target and provides support for the test suite author is called
test_server
. This is described in the chapter about writing test cases in this user's guide, and in the reference manual for thetest_server
module.
- The controlling part, which provides the low level operator interface, starts and stops the target node (if remote target) and slave nodes and writes log files, is called
test_server_ctrl
. The Test Server Controller should not be used directly when running tests. Instead a framework built on top of it should be used. The test server frameworkts
is such a framework, used when testing OTP. More information about how to write your own test server framework can be found in this user's guide and in the reference manual for thetest_server_ctrl
module.
- The test server framework
ts
provides the high level operator interface on top oftest_server_ctrl
. It is highly recommended to usets
instead of usingtest_server_ctrl
directly.ts
adds features like automatic compilation of test suites and data directories, collection of log files in central directories and single command interface for running all available tests. More information about how to run tests using the test server framework can be found in this users guide and in the reference manual for thets
module.
1.2 Definition of terms
- conf(iguration) case
- This is a group of test cases which need some specific configuration. A conf case contains an initiation function which sets up a specific configuration, one or more test cases using this configuration, and a cleanup function which restores the configuration. A conf case is specified in a test specification like this:
{conf,InitFunc,ListOfCases,CleanupFunc}
- datadir
- Data directory for a test suite. This directory contains any files used by the test suite, e.g. additional erlang modules, c code or data files. If the data directory contains code which must be compiled before the test suite is run, it should also contain a makefile source called Makefile.src defining how to compile.
- documentation clause
- One of the function clauses in a test case. This clause shall return a list of strings describing what the test case tests.
- execution clause
- One of the function clauses in a test case. This clause implements the actual test case, i.e. calls the functions that shall be tested and checks results. The clause shall crash if it fails.
- major log file
- This is the test suites log file.
- Makefile.src
- This file is used by the test server framework to generate a makefile for a datadir. It contains some special characters which are replaced according to the platform currently tested.
- minor log file
- This is a separate log file for each test case.
- privdir
- Private directory for a test suite. This directory should be used when the test suite needs to write to files.
- skip case
- A test case which shall be skipped.
- specification clause
- One of the function clauses in a test case. This clause shall return an empty list, a test specification or
{skip,Reason}
. If an empty list is returned, it means that the test case shall be executed, and so it must also have an execution clause. Note that the specification clause is always executed on the controller node, i.e. not on the target node.- test case
- A single test included in a test suite. Typically it tests one function in a module or application. A test case is implemented as a function in a test suite module. The function can have three clauses, the documentation-, specification- and execution clause.
- test specification
- A specification of which test suites and test cases to run. There can be test specifications on three different levels in a test. The top level is a test specification file which roughly specifies what to test for a whole application. Then there is a test specification for each test suite returned from the
all(suite)
function in the suite. And there can also be a test specification returned from the specification clause of a test case.- test specification file
- This is a text file containing the test specification for an application. The file has the extension ".spec" or ".spec.Platform", where Platform is e.g. "vxworks" or "ose".
- test suite
- An erlang module containing a collection of test cases for a specific application or module.
- topcase
- The first "command" in a test specification file. This command contains the test specification, like this:
{topcase,TestSpecification}