9 Using Common Test for Large Scale Testing
9.1 General
Large scale automated testing requires running multiple independent test sessions in parallel. This is accomplished by running a number of Common Test nodes on one or more hosts, testing different target systems. Configuring, starting and controlling the test nodes independently can be a cumbersome operation. To aid this kind of automated large scale testing, CT offers a master test node component, CT Master, that handles central configuration and control in a system of distributed CT nodes.
The CT Master server runs on one dedicated Erlang node and uses distributed Erlang to communicate with any number of CT test nodes, each hosting a regular CT server. Test specifications are used as input to specify what to test on which test nodes, using what configuration.
The CT Master server writes progress information to HTML log files similarly to the regular CT server. The logs contain test statistics and links to the log files written by each independent CT server.
The CT master API is exported by the ct_master module.
9.2 Usage
CT Master requires all test nodes to be on the same network and share a common file system. As of this date, CT Master can not start test nodes automatically. The nodes must have been started in advance for CT Master to be able to start test sessions on them.
Tests are started by calling:
ct_master:run(TestSpecs) or ct_master:run(TestSpecs, InclNodes, ExclNodes)
TestSpecs is either the name of a test specification file (string) or a list of test specifications. In case of a list, the specifications will be handled (and the corresponding tests executed) in sequence. An element in a TestSpecs list can also be list of test specifications. The specifications in such a list will be merged into one combined specification prior to test execution. For example:
ct_master:run(["ts1","ts2",["ts3","ts4"]])
means first the tests specified by "ts1" will run, then the tests specified by "ts2" and finally the tests specified by both "ts3" and "ts4".
The InclNodes argument to run/3 is a list of node names. The run/3 function runs the tests in TestSpecs just like run/1 but will also take any test in TestSpecs that's not explicitly tagged with a particular node name and execute it on the nodes listed in InclNodes. By using run/3 this way it is possible to use any test specification, with or without node information, in a large scale test environment! ExclNodes is a list of nodes that should be excluded from the test. I.e. tests that have been specified in the test specification to run on a particular node will not be performed if that node is at runtime listed in ExclNodes.
If CT Master fails initially to connect to any of the test nodes specified in a test specification or in the InclNodes list, the operator will be prompted with the option to either start over again (after manually checking the status of the node(s) in question), to run without the missing nodes, or to abort the operation.
When tests start, CT Master prints information to console about the nodes that are involved. CT Master also reports when tests finish, successfully or unsuccessfully. If connection is lost to a node, the test on that node is considered finished. CT Master will not attempt to reestablish contact with the failing node. At any time to get the current status of the test nodes, call the function:
ct_master:progress()
To stop one or more tests, use:
ct_master:abort() (stop all) or ct_master:abort(Nodes)
For detailed information about the ct_master API, please see the manual page for this module.
9.3 Test Specifications
The test specifications used as input to CT Master are fully compatible with the specifications used as input to the regular CT server. The syntax is described in the Running Test Suites chapter.
All test specification terms can have a NodeRefs element. This element specifies which node or nodes a configuration operation or a test is to be executed on. NodeRefs is defined as:
NodeRefs = all_nodes | [NodeRef] | NodeRef
where
NodeRef = NodeAlias | node() | master
A NodeAlias (atom()) is used in a test specification as a reference to a node name (so the actual node name only needs to be declared once). The alias is declared with a node term:
{node, NodeAlias, NodeName}
If NodeRefs has the value all_nodes, the operation or test will be performed on all given test nodes. (Declaring a term without a NodeRefs element actually has the same effect). If NodeRefs has the value master, the operation is only performed on the CT Master node (namely set the log directory or install an event handler).
Consider the example in the Running Test Suites chapter, now extended with node information and intended to be executed by the CT Master:
{node, node1, ct_node@host_x}.
{node, node2, ct_node@host_y}.
{logdir, master, "/home/test/master_logs"}.
{logdir, "/home/test/logs"}.
{config, node1, "/home/test/t1/cfg/config.cfg"}.
{config, node2, "/home/test/t2/cfg/config.cfg"}.
{config, "/home/test/t3/cfg/config.cfg"}.
{alias, t1, "/home/test/t1"}.
{alias, t2, "/home/test/t2"}.
{alias, t3, "/home/test/t3"}.
{suites, node1, t1, all}.
{skip_suites, node1, t1, [t1B_SUITE,t1D_SUITE], "Not implemented"}.
{skip_cases, node1, t1, t1A_SUITE, [test3,test4], "Irrelevant"}.
{skip_cases, node1, t1, t1C_SUITE, [test1], "Ignore"}.
{suites, node2, t2, [t2B_SUITE,t2C_SUITE]}.
{cases, node2, t2, t2A_SUITE, [test4,test1,test7]}.
{skip_suites, t3, all, "Not implemented"}.
This example specifies the same tests as the original example. But now if started with a call to ct_master:run(TestSpecName), the t1 test will be executed on node ct_node@host_x (node1), the t2 test on ct_node@host_y (node2) and the t3 test on both node1 and node2. The t1 config file will only be read on node1 and the t2 config file only on node2, while the t3 config file will be read on both node1 and node2. Both test nodes will write log files to the same directory. (The CT Master node will however use a different log directory than the test nodes).
If the test session is instead started with a call to ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x]), the result is that the t1 test does not run on ct_node@host_x (or any other node) while the t3 test runs on ct_node@host_y and ct_node@host_z.
A nice feature is that a test specification that includes node information can still be used as input to the regular Common Test server (as described in the Running Test Suites chapter). The result is that any test specified to run on a node with the same name as the Common Test node in question (typically ct@somehost if started with the run_test script), will be performed. Tests without explicit node association will always be performed too of course!
It is recommended that absolute paths are used for log directories, config files and test directory aliases in the test specifications so that current working directory settings are not important.