Common Test

User's Guide

Version 1.16

• User's Guide
• Reference Manual
• Release Notes
• PDF
• Top

• Expand All
• Contract All

Chapters

• Introduction
  • Top of chapter
  • Scope
  • Prerequisites
• Common Test Basics
  • Top of chapter
  • General
  • Test Suite Organisation
  • Support Libraries
  • Suites and Test Cases
  • External Interfaces
• Getting Started
  • Top of chapter
  • Introduction for Newcomers
  • Test Case Execution
  • A Simple Test Suite
  • A Test Suite with Configuration Functions
  • Questions and Answers
• Installation
  • Top of chapter
  • General Information
• Writing Test Suites
  • Top of chapter
  • Support for Test Suite Authors
  • Test Suites
  • Init and End per Suite
  • Init and End per Test Case
  • Test Cases
  • Test Case Information Function
  • Test Suite Information Function
  • Test Case Groups
  • Parallel Property and Nested Groups
  • Parallel Test Cases and I/O
  • Repeated Groups
  • Shuffled Test Case Order
  • Group Information Function
  • Information Functions for Init- and End-Configuration
  • Data and Private Directories
  • Execution Environment
  • Timetrap Time-Outs
  • Logging - Categories and Verbosity Levels
  • Illegal Dependencies
• Test Structure
  • Top of chapter
  • General
  • Skipping Test Cases
  • Definition of Terms
• Examples and Templates
  • Top of chapter
  • Test Suite Example
  • Test Suite Templates
• Running Tests and Analyzing Results
  • Top of chapter
  • Using the Common Test Framework
  • Automatic Compilation of Test Suites and Help Modules
  • Running Tests from the OS Command Line
  • Running Tests from the Erlang Shell or from an Erlang Program
  • Test Case Group Execution
  • Running the Interactive Shell Mode
  • Step-by-Step Execution of Test Cases with the Erlang Debugger
  • Test Specifications
  • Running Tests from the Web-Based GUI
  • Log Files
  • HTML Style Sheets
  • Repeating Tests
  • Silent Connections
• External Configuration Data
  • Top of chapter
  • General
  • Syntax
  • Requiring and Reading Configuration Data
  • Using Configuration Variables Defined in Multiple Files
  • Encrypted Configuration Files
  • Opening Connections Using Configuration Data
  • User-Specific Configuration Data Formats
  • Examples of Configuration Data Handling
  • Example of User-Specific Configuration Handler
• Code Coverage Analysis
  • Top of chapter
  • General
  • Use
  • Stopping the Cover Tool When Tests Are Completed
  • The Cover Specification File
  • Cross Cover Analysis
  • Logging
• Using Common Test for Large-Scale Testing
  • Top of chapter
  • General
  • Use
  • Test Specifications
  • Automatic Startup of Test Target Nodes
• Event Handling
  • Top of chapter
  • General
  • Use
• Dependencies between Test Cases and Suites
  • Top of chapter
  • General
  • Saving Configuration Data
  • Sequences
• Common Test Hooks
  • Top of chapter
  • General
  • Installing a CTH
  • CTH Scope
  • Manipulating Tests
  • Synchronizing External User Applications with Common Test
  • Example CTH
  • Built-In CTHs
• Some Thoughts about Testing
  • Top of chapter
  • Goals
  • What to Test

2 Common Test Basics

2.1  General

The Common Test framework is a tool that supports implementation and automated execution of test cases to any types of target systems. Common Test is the main tool being used in all testing- and verification activities that are part of Erlang/OTP system development and maintenance.

Test cases can be executed individually or in batches. Common Test also features a distributed testing mode with central control and logging. With this feature, multiple systems can be tested independently in one common session. This is useful, for example, when running automated large-scale regression tests.

The System Under Test (SUT) can consist of one or more target nodes. Common Test contains a generic test server that, together with other test utilities, is used to perform test case execution. The tests can be started from a GUI, from the OS shell, or from an Erlang shell. Test suites are files (Erlang modules) that contain the test cases (Erlang functions) to be executed. Support modules provide functions that the test cases use to do the tests.

In a black-box testing scenario, Common Test-based test programs connect to the target system(s) through standard O&M and CLI protocols. Common Test provides implementations of, and wrapper interfaces to, some of these protocols (most of which exist as standalone components and applications in OTP). The wrappers simplify configuration and add verbosity for logging purposes. Common Test is continously extended with useful support modules. However, notice that it is a straightforward task to use any Erlang/OTP component for testing purposes with Common Test, without needing a Common Test wrapper for it. It is as simple as calling Erlang functions. A number of target-independent interfaces are supported in Common Test, such as Generic Telnet and FTP. These can be specialized or used directly for controlling instruments, traffic load generators, and so on.

Common Test is also a very useful tool for white-box testing Erlang code (for example, module testing), as the test programs can call exported Erlang functions directly. There is very little overhead required for implementing basic test suites and executing simple tests. For black-box testing Erlang software, Erlang RPC and standard O&M interfaces can be used for example.

A test case can handle several connections to one or more target systems, instruments, and traffic generators in parallel to perform the necessary actions for a test. The handling of many connections in parallel is one of the major strengths of Common Test, thanks to the efficient support for concurrency in the Erlang runtime system, which Common Test users can take great advantage of.

2.2  Test Suite Organisation

Test suites are organized in test directories and each test suite can have a separate data directory. Typically, these files and directories are version-controlled similar to other forms of source code (possibly by a version control system like GIT or Subversion). However, Common Test does not itself put any requirements on (or has any awareness of) possible file and directory versions.

2.3  Support Libraries

Support libraries contain functions that are useful for all test suites, or for test suites in a specific functional area or subsystem. In addition to the general support libraries provided by the Common Test framework, and the various libraries and applications provided by Erlang/OTP, there can also be a need for customized (user specific) support libraries.

2.4  Suites and Test Cases

Testing is performed by running test suites (sets of test cases) or individual test cases. A test suite is implemented as an Erlang module named <suite_name>_SUITE.erl which contains a number of test cases. A test case is an Erlang function that tests one or more things. The test case is the smallest unit that the Common Test test server deals with.

Subsets of test cases, called test case groups, can also be defined. A test case group can have execution properties associated with it. Execution properties specify if the test cases in the group are to be executed in random order, in parallel, or in sequence, and if the execution of the group is to be repeated. Test case groups can also be nested (that is, a group can, besides test cases, contain subgroups).

Besides test cases and groups, the test suite can also contain configuration functions. These functions are meant to be used for setting up (and verifying) environment and state in the SUT (and/or the Common Test host node), required for the tests to execute correctly. Examples of operations are: Opening a connection to the SUT, initializing a database, running an installation script, and so on. Configuration can be performed per suite, per test case group, and per individual test case.

The test suite module must conform to a callback interface specified by the Common Test test server. For details, see section Writing Test Suites.

A test case is considered successful if it returns to the caller, no matter what the returned value is. However, a few return values have special meaning as follows:

• {skip,Reason} indicates that the test case is skipped.
• {comment,Comment} prints a comment in the log for the test case.
• {save_config,Config} makes the Common Test test server pass Config to the next test case.

A test case failure is specified as a runtime error (a crash), no matter what the reason for termination is. If you use Erlang pattern matching effectively, you can take advantage of this property. The result is concise and readable test case functions that look much more like scripts than actual programs. A simple example:

 session(_Config) ->
     {started,ServerId} = my_server:start(),
     {clients,[]} = my_server:get_clients(ServerId),
     MyId = self(),
     connected = my_server:connect(ServerId, MyId),
     {clients,[MyId]} = my_server:get_clients(ServerId),
     disconnected = my_server:disconnect(ServerId, MyId),
     {clients,[]} = my_server:get_clients(ServerId),
     stopped = my_server:stop(ServerId).

As a test suite runs, all information (including output to stdout) is recorded in many different log files. A minimum of information is displayed in the user console (only start and stop information, plus a note for each failed test case).

The result from each test case is recorded in a dedicated HTML log file, created for the particular test run. An overview page displays each test case represented by a table row showing total execution time, if the case was successful, failed, or skipped, plus an optional user comment. For a failed test case, the reason for termination is also printed in the comment field. The overview page has a link to each test case log file, providing simple navigation with any standard HTML browser.

2.5  External Interfaces

The Common Test test server requires that the test suite defines and exports the following mandatory or optional callback functions:

all()

Returns a list of all test cases and groups in the suite. (Mandatory)

suite()

Information function used to return properties for the suite. (Optional)

groups()

For declaring test case groups. (Optional)

init_per_suite(Config)

Suite level configuration function, executed before the first test case. (Optional)

end_per_suite(Config)

Suite level configuration function, executed after the last test case. (Optional)

group(GroupName)

Information function used to return properties for a test case group. (Optional)

init_per_group(GroupName, Config)

Configuration function for a group, executed before the first test case. (Optional)

end_per_group(GroupName, Config)

Configuration function for a group, executed after the last test case. (Optional)

init_per_testcase(TestCase, Config)

Configuration function for a testcase, executed before each test case. (Optional)

end_per_testcase(TestCase, Config)

Configuration function for a testcase, executed after each test case. (Optional)

For each test case, the Common Test test server expects the following functions:

Testcasename()

Information function that returns a list of test case properties. (Optional)

Testcasename(Config)

The test case function.