Common Test
User's Guide
Version 1.5.5

Chapters

• Common Test Basics
  • Top of chapter
  • Introduction
  • Test Suite Organisation
  • Support Libraries
  • Suites and Test Cases
  • External Interfaces
• Installation
  • Top of chapter
  • General information
• Writing Test Suites
  • Top of chapter
  • Support for test suite authors
  • Test suites
  • Init and end per suite
  • Init and end per test case
  • Test cases
  • Test case info function
  • Test suite info function
  • Test case groups
  • The parallel property and nested groups
  • Repeated groups
  • Shuffled test case order
  • Data and Private Directories
  • Execution environment
  • Timetrap timeouts
  • Illegal dependencies
• Test Structure
  • Top of chapter
  • Test structure
  • Skipping test cases
  • Definition of terms
• Examples and Templates
  • Top of chapter
  • Test suite example
  • Test suite templates
• Running Tests
  • Top of chapter
  • Using the Common Test Framework
  • Automatic compilation of test suites and help modules
  • Running tests from the OS command line
  • Running tests from the Web based GUI
  • Running tests from the Erlang shell or from an Erlang program
  • Running the interactive shell mode
  • Step by step execution of test cases with the Erlang Debugger
  • Using test specifications
  • Log files
  • HTML Style Sheets
  • Repeating tests
  • Silent Connections
• External Configuration Data
  • Top of chapter
  • General
  • Syntax
  • Requiring and reading configuration data
  • Using configuration variables defined in multiple files
  • Encrypted configuration files
  • Opening connections by using configuration data
  • User specific configuration data formats
  • Examples of configuration data handling
  • Example of user specific configuration handler
• Code Coverage Analysis
  • Top of chapter
  • General
  • Usage
  • The cover specification file
  • Logging
• Using Common Test for Large Scale Testing
  • Top of chapter
  • General
  • Usage
  • Test Specifications
  • Automatic startup of test target nodes
• Event Handling
  • Top of chapter
  • General
  • Usage
• Dependencies between Test Cases and Suites
  • Top of chapter
  • General
  • Saving configuration data
  • Sequences
• Common Test Hooks
  • Top of chapter
  • General
  • Installing a CTH
  • CTH Scope
  • Manipulating tests
  • Example CTH
• Some thoughts about testing
  • Top of chapter
  • Goals
  • What to test?

1 Common Test Basics

1.1  Introduction

The Common Test framework (CT) is a tool which can support implementation and automated execution of test cases towards different types of target systems. The framework is based on the OTP Test Server. Test cases can be run individually or in batches. Common Test also features a distributed testing mode with central control and logging. This feature makes it possible to test multiple systems independently in one common session. This can be very useful e.g. for running automated large-scale regression tests.

The SUT (System Under Test) may consist of one or several target nodes. CT contains a generic test server which together with other test utilities is used to perform test case execution. It is possible to start the tests from the CT GUI or from an OS- or Erlang shell prompt. Test suites are files (Erlang modules) that contain the test cases (Erlang functions) to be executed. Support modules provide functions that the test cases utilize in order to carry out the tests.

The main idea is that CT based test programs connect to the target system(s) via standard O&M interfaces. CT provides implementations and wrappers of some of these O&M interfaces and will be extended with more interfaces later. There are a number of target independent interfaces supported in CT such as Generic Telnet, FTP etc. which can be specialized or used directly for controlling instruments, traffic generators etc.

Common Test is also a very useful tool for white-box testing Erlang code since the test programs can call Erlang API functions directly. For black-box testing Erlang software, Erlang RPC as well as standard O&M interfaces can be used.

A test case can handle several connections towards one or several target systems, instruments and traffic generators in parallel in order to perform the necessary actions for a test. The handling of many connections in parallel is one of the major strengths of Common Test!

1.2  Test Suite Organisation

The test suites are organized in test directories and each test suite may have a separate data directory. Typically, these files and directories are version controlled similarly to other forms of source code (possibly by means of a version control system like GIT or Subversion). However, CT does not itself put any requirements on (or has any form of awareness of) possible file and directory versions.

1.3  Support Libraries

Support libraries contain functions that are useful for all test suites, or for test suites in a specific functional area or subsystem. In addition to the general support libraries provided by the CT framework, and the various libraries and applications provided by Erlang/OTP, there might also be a need for customized (user specific) support libraries.

1.4  Suites and Test Cases

Testing is performed by running test suites (sets of test cases) or individual test cases. A test suite is implemented as an Erlang module named <suite_name>_SUITE.erl which contains a number of test cases. A test case is an Erlang function which tests one or more things. The test case is the smallest unit that the CT test server deals with.

Subsets of test cases, called test case groups, may also be defined. A test case group can have execution properties associated with it. Execution properties specify whether the test cases in the group should be executed in random order, in parallel, in sequence, and if the execution of the group should be repeated. Test case groups may also be nested (i.e. a group may, besides test cases, contain sub-groups).

Besides test cases and groups, the test suite may also contain configuration functions. These functions are meant to be used for setting up (and verifying) environment and state on the SUT (and/or the CT host node), required for the tests to execute correctly. Examples of operations: Opening a connection to the SUT, initializing a database, running an installation script, etc. Configuration may be performed per suite, per test case group and per individual test case.

The test suite module must conform to a callback interface specified by the CT test server. See the Writing Test Suites chapter for more information.

A test case is considered successful if it returns to the caller, no matter what the returned value is. A few return values have special meaning however (such as {skip,Reason} which indicates that the test case is skipped, {comment,Comment} which prints a comment in the log for the test case and {save_config,Config} which makes the CT test server pass Config to the next test case). A test case failure is specified as a runtime error (a crash), no matter what the reason for termination is. If you use Erlang pattern matching effectively, you can take advantage of this property. The result will be concise and readable test case functions that look much more like scripts than actual programs. Simple example:

      session(_Config) ->
          {started,ServerId} = my_server:start(),
          {clients,[]} = my_server:get_clients(ServerId),
          MyId = self(),
          connected = my_server:connect(ServerId, MyId),
          {clients,[MyId]} = my_server:get_clients(ServerId),
          disconnected = my_server:disconnect(ServerId, MyId),
          {clients,[]} = my_server:get_clients(ServerId),
          stopped = my_server:stop(ServerId).
    

As a test suite runs, all information (including output to stdout) is recorded in several different log files. A minimum of information is displayed in the user console (only start and stop information, plus a note for each failed test case).

The result from each test case is recorded in a dedicated HTML log file, created for the particular test run. An overview page displays each test case represented by row in a table showing total execution time, whether the case was successful, failed or skipped, plus an optional user comment. (For a failed test case, the reason for termination is also printed in the comment field). The overview page has a link to each test case log file, providing simple navigation with any standard HTML browser.

1.5  External Interfaces

The CT test server requires that the test suite defines and exports the following mandatory or optional callback functions:

all()

Returns a list of all test cases in the suite. (Mandatory)

suite()

Info function used to return properties for the suite. (Optional)

groups()

For declaring test case groups. (Optional)

init_per_suite(Config)

Suite level configuration function, executed before the first test case. (Optional)

end_per_suite(Config)

Suite level configuration function, executed after the last test case. (Optional)

init_per_group(GroupName, Config)

Configuration function for a group, executed before the first test case. (Mandatory if groups are defined)

end_per_group(GroupName, Config)

Configuration function for a group, executed after the last test case. (Mandatory if groups are defined)

init_per_testcase(TestCase, Config)

Configuration function for a testcase, executed before each test case. (Optional)

end_per_testcase(TestCase, Config)

Configuration function for a testcase, executed after each test case. (Optional)

For each test case the CT test server expects these functions:

Testcasename()

Info function that returns a list of test case properties. (Optional)

Testcasename(Config)

The actual test case function.