1 Test Server Basics

1.1 Introduction

Test Server is a portable test tool for automated testing of Erlang programs and OTP applications. It provides an interface for running test programs directly with Test Server as well as an interface for integrating Test Server with a framework application. The latter makes it possible to use Test Server as the engine of a higher level test tool application.

It is strongly recommended that Test Server be used from inside a framework application, rather than interfaced directly for running test programs. Test Server can be pretty difficult to use since it's a very general and quite extensive and complex application. Furthermore, the test_server_ctrl functions are not meant to be used from within the actual test programs. The framework should handle communication with Test Server and deal with the more complex aspects of this interaction automatically so that a higher level interface may be provided for the tester. For test tool usage to be productive, a simpler, more intuitive and (if required) more specific interface is required than what Test Server can provide.

OTP delivers a general purpose framework for Test Server, called Common Test. This application is a tool well suited for automated black box testing of target systems of any kind (not necessarily implemented in Erlang). Common Test is also a very useful tool for white box testing of Erlang programs and OTP applications. Unless a more specific functionality and/or user interface is required (in which case you might need to implement your own framework), Common Test should do the job for you. Please read the Common Test User's Guide and reference manual for more information.

Under normal circumstances, knowledge about the Test Server application is not required for using the Common Test framework. However, if you want to use Test Server without a framework, or learn how to integrate it with your own framework, please read on...

1.2 Getting started

Testing when using Test Server is done by running test suites. A test suite is a number of test cases, where each test case tests one or more things. The test case is the smallest unit that the test server deals with. One or more test cases are grouped together into one ordinary Erlang module, which is called a test suite. Several test suite modules can be grouped together in special test specification files representing whole application and/or system test "jobs".

The test suite Erlang module must follow a certain interface, which is specified by Test Server. See the section on writing test suites for details about this.

Each test case is considered a success if it returns to the caller, no matter what the returned value is. An exception to this is the return value {skip, Reason} which indicates that the test case is skipped. A failure is specified as a crash, no matter what the crash reason is.

As a test suite runs, all information (including output to stdout) is recorded in several different log files. A minimum of information is displayed to the user console. This only include start and stop information, plus a note for each failed test case.

The result from each test case is recorded in an HTML log file which is created for each test run. Every test case gets one row in a table presenting total time, whether the case was successful or not, if it was skipped, and possibly also a comment. The HTML file has links to each test case's logfile, which may be viewed from e.g. Netscape or any other HTML capable browser.

The Test Server consists of three parts:

The part that executes the test suites on target and provides support for the test suite author is called test_server. This is described in the chapter about writing test cases in this user's guide, and in the reference manual for the test_server module.
The controlling part, which provides the low level operator interface, starts and stops the target node (if remote target) and slave nodes and writes log files, is called test_server_ctrl. The Test Server Controller should not be used directly when running tests. Instead a framework built on top of it should be used. More information about how to write your own framework can be found in this user's guide and in the reference manual for the test_server_ctrl module.

1.3 Definition of terms

conf(iguration) case: This is a group of test cases which need some specific configuration. A conf case contains an initiation function which sets up a specific configuration, one or more test cases using this configuration, and a cleanup function which restores the configuration. A conf case is specified in a test specification either like this:{conf,InitFunc,ListOfCases,CleanupFunc}, or this: {conf,Properties,InitFunc,ListOfCases,CleanupFunc}
datadir: Data directory for a test suite. This directory contains any files used by the test suite, e.g. additional erlang modules, c code or data files. If the data directory contains code which must be compiled before the test suite is run, it should also contain a makefile source called Makefile.src defining how to compile.
documentation clause: One of the function clauses in a test case. This clause shall return a list of strings describing what the test case tests.
execution clause: One of the function clauses in a test case. This clause implements the actual test case, i.e. calls the functions that shall be tested and checks results. The clause shall crash if it fails.
major log file: This is the test suites log file.
Makefile.src: This file is used by the test server framework to generate a makefile for a datadir. It contains some special characters which are replaced according to the platform currently tested.
minor log file: This is a separate log file for each test case.
privdir: Private directory for a test suite. This directory should be used when the test suite needs to write to files.
skip case: A test case which shall be skipped.
specification clause: One of the function clauses in a test case. This clause shall return an empty list, a test specification or {skip,Reason}. If an empty list is returned, it means that the test case shall be executed, and so it must also have an execution clause. Note that the specification clause is always executed on the controller node, i.e. not on the target node.
test case: A single test included in a test suite. Typically it tests one function in a module or application. A test case is implemented as a function in a test suite module. The function can have three clauses, the documentation-, specification- and execution clause.
test specification: A specification of which test suites and test cases to run. There can be test specifications on three different levels in a test. The top level is a test specification file which roughly specifies what to test for a whole application. Then there is a test specification for each test suite returned from the all(suite) function in the suite. And there can also be a test specification returned from the specification clause of a test case.
test specification file: This is a text file containing the test specification for an application. The file has the extension ".spec" or ".spec.Platform", where Platform is e.g. "vxworks".
test suite: An erlang module containing a collection of test cases for a specific application or module.
topcase: The first "command" in a test specification file. This command contains the test specification, like this: {topcase,TestSpecification}