Common Test
User's Guide
Version 1.11

Chapters

• Common Test Basics
  • Top of chapter
  • Introduction
  • Test Suite Organisation
  • Support Libraries
  • Suites and Test Cases
  • External Interfaces
• Getting Started
  • Top of chapter
  • Are you new around here?
  • Test case execution
  • A simple test suite
  • A test suite with configuration functions
  • What happens next?
• Installation
  • Top of chapter
  • General information
• Writing Test Suites
  • Top of chapter
  • Support for test suite authors
  • Test suites
  • Init and end per suite
  • Init and end per test case
  • Test cases
  • Test case info function
  • Test suite info function
  • Test case groups
  • The parallel property and nested groups
  • Parallel test cases and IO
  • Repeated groups
  • Shuffled test case order
  • Group info function
  • Info functions for init- and end-configuration
  • Data and Private Directories
  • Execution environment
  • Timetrap timeouts
  • Logging - categories and verbosity levels
  • Illegal dependencies
• Test Structure
  • Top of chapter
  • Test structure
  • Skipping test cases
  • Definition of terms
• Examples and Templates
  • Top of chapter
  • Test suite example
  • Test suite templates
• Running Tests and Analyzing Results
  • Top of chapter
  • Using the Common Test Framework
  • Automatic compilation of test suites and help modules
  • Running tests from the OS command line
  • Running tests from the Erlang shell or from an Erlang program
  • Test case group execution
  • Running the interactive shell mode
  • Step by step execution of test cases with the Erlang Debugger
  • Test Specifications
  • Running tests from the Web based GUI
  • Log files
  • HTML Style Sheets
  • Repeating tests
  • Silent Connections
• External Configuration Data
  • Top of chapter
  • General
  • Syntax
  • Requiring and reading configuration data
  • Using configuration variables defined in multiple files
  • Encrypted configuration files
  • Opening connections by using configuration data
  • User specific configuration data formats
  • Examples of configuration data handling
  • Example of user specific configuration handler
• Code Coverage Analysis
  • Top of chapter
  • General
  • Usage
  • Stopping the cover tool when tests are completed
  • The cover specification file
  • Cross cover analysis
  • Logging
• Using Common Test for Large Scale Testing
  • Top of chapter
  • General
  • Usage
  • Test Specifications
  • Automatic startup of test target nodes
• Event Handling
  • Top of chapter
  • General
  • Usage
• Dependencies between Test Cases and Suites
  • Top of chapter
  • General
  • Saving configuration data
  • Sequences
• Common Test Hooks
  • Top of chapter
  • General
  • Installing a CTH
  • CTH Scope
  • Manipulating tests
  • Synchronizing external user applications with Common Test
  • Example CTH
  • Built-in CTHs
• Some thoughts about testing
  • Top of chapter
  • Goals
  • What to test?

10 Using Common Test for Large Scale Testing

10.1  General

Large scale automated testing requires running multiple independent test sessions in parallel. This is accomplished by running a number of Common Test nodes on one or more hosts, testing different target systems. Configuring, starting and controlling the test nodes independently can be a cumbersome operation. To aid this kind of automated large scale testing, CT offers a master test node component, CT Master, that handles central configuration and control in a system of distributed CT nodes.

The CT Master server runs on one dedicated Erlang node and uses distributed Erlang to communicate with any number of CT test nodes, each hosting a regular CT server. Test specifications are used as input to specify what to test on which test nodes, using what configuration.

The CT Master server writes progress information to HTML log files similarly to the regular CT server. The logs contain test statistics and links to the log files written by each independent CT server.

The CT master API is exported by the ct_master module.

10.2  Usage

CT Master requires all test nodes to be on the same network and share a common file system. As of this date, CT Master can not start test nodes automatically. The nodes must have been started in advance for CT Master to be able to start test sessions on them.

Tests are started by calling:

ct_master:run(TestSpecs) or ct_master:run(TestSpecs, InclNodes, ExclNodes)

TestSpecs is either the name of a test specification file (string) or a list of test specifications. In case of a list, the specifications will be handled (and the corresponding tests executed) in sequence. An element in a TestSpecs list can also be list of test specifications. The specifications in such a list will be merged into one combined specification prior to test execution. For example:

ct_master:run(["ts1","ts2",["ts3","ts4"]])

means first the tests specified by "ts1" will run, then the tests specified by "ts2" and finally the tests specified by both "ts3" and "ts4".

The InclNodes argument to run/3 is a list of node names. The run/3 function runs the tests in TestSpecs just like run/1 but will also take any test in TestSpecs that's not explicitly tagged with a particular node name and execute it on the nodes listed in InclNodes. By using run/3 this way it is possible to use any test specification, with or without node information, in a large scale test environment! ExclNodes is a list of nodes that should be excluded from the test. I.e. tests that have been specified in the test specification to run on a particular node will not be performed if that node is at runtime listed in ExclNodes.

If CT Master fails initially to connect to any of the test nodes specified in a test specification or in the InclNodes list, the operator will be prompted with the option to either start over again (after manually checking the status of the node(s) in question), to run without the missing nodes, or to abort the operation.

When tests start, CT Master prints information to console about the nodes that are involved. CT Master also reports when tests finish, successfully or unsuccessfully. If connection is lost to a node, the test on that node is considered finished. CT Master will not attempt to reestablish contact with the failing node. At any time to get the current status of the test nodes, call the function:

ct_master:progress()

To stop one or more tests, use:

ct_master:abort() (stop all) or ct_master:abort(Nodes)

For detailed information about the ct_master API, please see the manual page for this module.

10.3  Test Specifications

The test specifications used as input to CT Master are fully compatible with the specifications used as input to the regular CT server. The syntax is described in the Running Test Suites chapter.

All test specification terms can have a NodeRefs element. This element specifies which node or nodes a configuration operation or a test is to be executed on. NodeRefs is defined as:

NodeRefs = all_nodes | [NodeRef] | NodeRef

where

NodeRef = NodeAlias | node() | master

A NodeAlias (atom()) is used in a test specification as a reference to a node name (so the actual node name only needs to be declared once, which can of course also be achieved using constants). The alias is declared with a node term:

{node, NodeAlias, NodeName}

If NodeRefs has the value all_nodes, the operation or test will be performed on all given test nodes. (Declaring a term without a NodeRefs element actually has the same effect). If NodeRefs has the value master, the operation is only performed on the CT Master node (namely set the log directory or install an event handler).

Consider the example in the Running Test Suites chapter, now extended with node information and intended to be executed by the CT Master:

      {define, 'Top', "/home/test"}.
      {define, 'T1', "'Top'/t1"}.
      {define, 'T2', "'Top'/t2"}.
      {define, 'T3', "'Top'/t3"}.
      {define, 'CfgFile', "config.cfg"}.
      {define, 'Node', ct_node}.

      {node, node1, 'Node@host_x'}.
      {node, node2, 'Node@host_y'}.

      {logdir, master, "'Top'/master_logs"}.
      {logdir, "'Top'/logs"}.
      
      {config, node1, "'T1'/'CfgFile'"}.
      {config, node2, "'T2'/'CfgFile'"}.
      {config, "'T3'/'CfgFile'"}.
      
      {suites, node1, 'T1', all}.
      {skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
      {skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
      {skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}.
      
      {suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}.
      {cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}.
      
      {skip_suites, 'T3', all, "Not implemented"}.

This example specifies the same tests as the original example. But now if started with a call to ct_master:run(TestSpecName), the t1 test will be executed on node ct_node@host_x (node1), the t2 test on ct_node@host_y (node2) and the t3 test on both node1 and node2. The t1 config file will only be read on node1 and the t2 config file only on node2, while the t3 config file will be read on both node1 and node2. Both test nodes will write log files to the same directory. (The CT Master node will however use a different log directory than the test nodes).

If the test session is instead started with a call to ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x]), the result is that the t1 test does not run on ct_node@host_x (or any other node) while the t3 test runs on ct_node@host_y and ct_node@host_z.

A nice feature is that a test specification that includes node information can still be used as input to the regular Common Test server (as described in the Running Test Suites chapter). The result is that any test specified to run on a node with the same name as the Common Test node in question (typically ct@somehost if started with the ct_run program), will be performed. Tests without explicit node association will always be performed too of course!

10.4  Automatic startup of test target nodes

It is possible to automatically start, and perform initial actions, on test target nodes by using the test specification term init.

Currently, two sub-terms are supported, node_start and eval.

Example:

     {node, node1, node1@host1}.
     {node, node2, node1@host2}.
     {node, node3, node2@host2}.
     {node, node4, node1@host3}.
     {init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}.
     {init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}.
     {init, node4, {eval, {module, function, []}}}.

This test specification declares that node1@host1 is to be started using the user callback function callback_module:my_slave_callback/0, and nodes node1@host2 and node2@host2 will be started with the default callback module ct_slave. The given user name and password is used to log into remote host host2. Also, the function module:function/0 will be evaluated on node1@host3, and the result of this call will be printed to the log.

The default ct_slave callback module, which is part of the Common Test application, has the following features:

• Starting Erlang target nodes on local or remote hosts (ssh is used for communication).
• Ability to start an Erlang emulator with additional flags (any flags supported by erl are supported).
• Supervision of a node being started by means of internal callback functions. Used to prevent hanging nodes. (Configurable).
• Monitoring of the master node by the slaves. A slave node may be stopped in case the master node terminates. (Configurable).
• Execution of user functions after a slave node is started. Functions can be given as a list of {Module, Function, Arguments} tuples.

Note that it is possible to specify an eval term for the node as well as startup_functions in the node_start options list. In this case first the node will be started, then the startup_functions are executed, and finally functions specified with eval are called.