Common Test

User's Guide

Version 1.16

• User's Guide
• Reference Manual
• Release Notes
• PDF
• Top

• Expand All
• Contract All

Chapters

• Introduction
  • Top of chapter
  • Scope
  • Prerequisites
• Common Test Basics
  • Top of chapter
  • General
  • Test Suite Organisation
  • Support Libraries
  • Suites and Test Cases
  • External Interfaces
• Getting Started
  • Top of chapter
  • Introduction for Newcomers
  • Test Case Execution
  • A Simple Test Suite
  • A Test Suite with Configuration Functions
  • Questions and Answers
• Installation
  • Top of chapter
  • General Information
• Writing Test Suites
  • Top of chapter
  • Support for Test Suite Authors
  • Test Suites
  • Init and End per Suite
  • Init and End per Test Case
  • Test Cases
  • Test Case Information Function
  • Test Suite Information Function
  • Test Case Groups
  • Parallel Property and Nested Groups
  • Parallel Test Cases and I/O
  • Repeated Groups
  • Shuffled Test Case Order
  • Group Information Function
  • Information Functions for Init- and End-Configuration
  • Data and Private Directories
  • Execution Environment
  • Timetrap Time-Outs
  • Logging - Categories and Verbosity Levels
  • Illegal Dependencies
• Test Structure
  • Top of chapter
  • General
  • Skipping Test Cases
  • Definition of Terms
• Examples and Templates
  • Top of chapter
  • Test Suite Example
  • Test Suite Templates
• Running Tests and Analyzing Results
  • Top of chapter
  • Using the Common Test Framework
  • Automatic Compilation of Test Suites and Help Modules
  • Running Tests from the OS Command Line
  • Running Tests from the Erlang Shell or from an Erlang Program
  • Test Case Group Execution
  • Running the Interactive Shell Mode
  • Step-by-Step Execution of Test Cases with the Erlang Debugger
  • Test Specifications
  • Running Tests from the Web-Based GUI
  • Log Files
  • HTML Style Sheets
  • Repeating Tests
  • Silent Connections
• External Configuration Data
  • Top of chapter
  • General
  • Syntax
  • Requiring and Reading Configuration Data
  • Using Configuration Variables Defined in Multiple Files
  • Encrypted Configuration Files
  • Opening Connections Using Configuration Data
  • User-Specific Configuration Data Formats
  • Examples of Configuration Data Handling
  • Example of User-Specific Configuration Handler
• Code Coverage Analysis
  • Top of chapter
  • General
  • Use
  • Stopping the Cover Tool When Tests Are Completed
  • The Cover Specification File
  • Cross Cover Analysis
  • Logging
• Using Common Test for Large-Scale Testing
  • Top of chapter
  • General
  • Use
  • Test Specifications
  • Automatic Startup of Test Target Nodes
• Event Handling
  • Top of chapter
  • General
  • Use
• Dependencies between Test Cases and Suites
  • Top of chapter
  • General
  • Saving Configuration Data
  • Sequences
• Common Test Hooks
  • Top of chapter
  • General
  • Installing a CTH
  • CTH Scope
  • Manipulating Tests
  • Synchronizing External User Applications with Common Test
  • Example CTH
  • Built-In CTHs
• Some Thoughts about Testing
  • Top of chapter
  • Goals
  • What to Test

15 Some Thoughts about Testing

15.1  Goals

It is not possible to prove that a program is correct by testing. On the contrary, it has been formally proven that it is impossible to prove programs in general by testing. Theoretical program proofs or plain examination of code can be viable options for those wishing to certify that a program is correct. The test server, as it is based on testing, cannot be used for certification. Its intended use is instead to (cost effectively) find bugs. A successful test suite is one that reveals a bug. If a test suite results in OK, then we know very little that we did not know before.

15.2  What to Test

There are many kinds of test suites. Some concentrate on calling every function or command (in the documented way) in a certain interface. Some others do the same, but use all kinds of illegal parameters, and verify that the server stays alive and rejects the requests with reasonable error codes. Some test suites simulate an application (typically consisting of a few modules of an application), some try to do tricky requests in general, and some test suites even test internal functions with help of special Load Modules on target.

Another interesting category of test suites is the one checking that fixed bugs do not reoccur. When a bugfix is introduced, a test case that checks for that specific bug is written and submitted to the affected test suites.

Aim for finding bugs. Write whatever test that has the highest probability of finding a bug, now or in the future. Concentrate more on the critical parts. Bugs in critical subsystems are much more expensive than others.

Aim for functionality testing rather than implementation details. Implementation details change quite often, and the test suites are to be long lived. Implementation details often differ on different platforms and versions. If implementation details must be tested, try to factor them out into separate test cases. These test cases can later be rewritten or skipped.

Also, aim for testing everything once, no less, no more. It is not effective having every test case fail only because one function in the interface changed.