Dependencies between Test Cases and Suites

11 Dependencies between Test Cases and Suites

11.1 General

When creating test suites, it is strongly recommended to not create dependencies between test cases, i.e. letting test cases depend on the result of previous test cases. There are various reasons for this, for example:

It makes it impossible to run test cases individually.
It makes it impossible to run test cases in different order.
It makes debugging very difficult (since a fault could be the result of a problem in a different test case than the one failing).
There exists no good and explicit ways to declare dependencies, so it may be very difficult to see and understand these in test suite code and in test logs.
Extending, restructuring and maintaining test suites with test case dependencies is difficult.

There are often sufficient means to work around the need for test case dependencies. Generally, the problem is related to the state of the system under test (SUT). The action of one test case may alter the state of the system and for some other test case to run properly, the new state must be known.

Instead of passing data between test cases, it is recommended that the test cases read the state from the SUT and perform assertions (i.e. let the test case run if the state is as expected, otherwise reset or fail) and/or use the state to set variables necessary for the test case to execute properly. Common actions can often be implemented as library functions for test cases to call to set the SUT in a required state. (Such common actions may of course also be separately tested if necessary, to ensure they are working as expected). It is sometimes also possible, but not always desirable, to group tests together in one test case, i.e. let a test case perform a "scenario" test (a test that consists of subtests).

Consider for example a server application under test. The following functionality is to be tested:

Starting the server.
Configuring the server.
Connecting a client to the server.
Disconnecting a client from the server.
Stopping the server.

There are obvious dependencies between the listed functions. We can't configure the server if it hasn't first been started, we can't connect a client until the server has been properly configured, etc. If we want to have one test case for each of the functions, we might be tempted to try to always run the test cases in the stated order and carry possible data (identities, handles, etc) between the cases and therefore introduce dependencies between them. To avoid this we could consider starting and stopping the server for every test. We would implement the start and stop action as common functions that may be called from init_per_testcase and end_per_testcase. (We would of course test the start and stop functionality separately). The configuration could perhaps also be implemented as a common function, maybe grouped with the start function. Finally the testing of connecting and disconnecting a client may be grouped into one test case. The resulting suite would look something like this:

      -module(my_server_SUITE).
      -compile(export_all).
      -include_lib("ct.hrl").

      %%% init and end functions...

      suite() -> [{require,my_server_cfg}].

      init_per_testcase(start_and_stop, Config) ->
          Config;

      init_per_testcase(config, Config) ->
          [{server_pid,start_server()} | Config];

      init_per_testcase(_, Config) ->
          ServerPid = start_server(),
          configure_server(),
          [{server_pid,ServerPid} | Config].

      end_per_testcase(start_and_stop, _) ->
          ok;

      end_per_testcase(_, _) ->
          ServerPid = ?config(server_pid),
          stop_server(ServerPid).

      %%% test cases...

      all() -> [start_and_stop, config, connect_and_disconnect].

      %% test that starting and stopping works
      start_and_stop(_) ->
          ServerPid = start_server(),
          stop_server(ServerPid).

      %% configuration test
      config(Config) ->
          ServerPid = ?config(server_pid, Config),
          configure_server(ServerPid).

      %% test connecting and disconnecting client
      connect_and_disconnect(Config) ->
          ServerPid = ?config(server_pid, Config),
          {ok,SessionId} = my_server:connect(ServerPid),
          ok = my_server:disconnect(ServerPid, SessionId).

      %%% common functions...

      start_server() ->
          {ok,ServerPid} = my_server:start(),
          ServerPid.

      stop_server(ServerPid) ->
          ok = my_server:stop(),
          ok.

      configure_server(ServerPid) ->
          ServerCfgData = ct:get_config(my_server_cfg),
          ok = my_server:configure(ServerPid, ServerCfgData),
          ok.

11.2 Saving configuration data

There might be situations where it is impossible, or infeasible at least, to implement independent test cases. Maybe it is simply not possible to read the SUT state. Maybe resetting the SUT is impossible and it takes much too long to restart the system. In situations where test case dependency is necessary, CT offers a structured way to carry data from one test case to the next. The same mechanism may also be used to carry data from one test suite to the next.

The mechanism for passing data is called save_config. The idea is that one test case (or suite) may save the current value of Config - or any list of key-value tuples - so that it can be read by the next executing test case (or test suite). The configuration data is not saved permanently but can only be passed from one case (or suite) to the next.

To save Config data, return the tuple:

{save_config,ConfigList}

from end_per_testcase or from the main test case function. To read data saved by a previous test case, use the config macro with a saved_config key:

{Saver,ConfigList} = ?config(saved_config, Config)

Saver (atom()) is the name of the previous test case (where the data was saved). The config macro may be used to extract particular data also from the recalled ConfigList. It is strongly recommended that Saver is always matched to the expected name of the saving test case. This way problems due to restructuring of the test suite may be avoided. Also it makes the dependency more explicit and the test suite easier to read and maintain.

To pass data from one test suite to another, the same mechanism is used. The data should be saved by the end_per_suite function and read by init_per_suite in the suite that follows. When passing data between suites, Saver carries the name of the test suite.

Example:

        -module(server_b_SUITE).
        -compile(export_all).
        -include_lib("ct.hrl").

        %%% init and end functions...

        init_per_suite(Config) ->
            %% read config saved by previous test suite
            {server_a_SUITE,OldConfig} = ?config(saved_config, Config),
            %% extract server identity (comes from server_a_SUITE)
            ServerId = ?config(server_id, OldConfig),
            SessionId = connect_to_server(ServerId),
            [{ids,{ServerId,SessionId}} | Config].

        end_per_suite(Config) ->
            %% save config for server_c_SUITE (session_id and server_id)
            {save_config,Config}

        %%% test cases...

        all() -> [allocate, deallocate].

        allocate(Config) ->
            {ServerId,SessionId} = ?config(ids, Config),
            {ok,Handle} = allocate_resource(ServerId, SessionId),
            %% save handle for deallocation test
            NewConfig = [{handle,Handle}],
            {save_config,NewConfig}.

        deallocate(Config) ->
            {ServerId,SessionId} = ?config(ids, Config),
            {allocate,OldConfig} = ?config(saved_config, Config),
            Handle = ?config(handle, OldConfig),
            ok = deallocate_resource(ServerId, SessionId, Handle).

It is also possible to save Config data from a test case that is to be skipped. To accomplish this, return the following tuple:

{skip_and_save,Reason,ConfigList}

The result will be that the test case is skipped with Reason printed to the log file (as described in previous chapters), and ConfigList is saved for the next test case. ConfigList may be read by means of ?config(saved_config, Config), as described above. skip_and_save may also be returned from init_per_suite, in which case the saved data can be read by init_per_suite in the suite that follows.

11.3 Sequences

It is possible that test cases depend on each other so that if one case fails, the following test(s) should not be executed. Typically, if the save_config facility is used and a test case that is expected to save data crashes, the following case can not run. CT offers a way to declare such dependencies, called sequences.

A sequence of test cases is declared by means of the function sequences/0. This function should return a list of tuples on the format: {SeqTag,TestCases}. SeqTag is an atom that identifies one particular sequence. TestCases is a list of test case names. Sequences must be included in the list that all/0 returns. They are declared as: {sequence,SeqTag}.

For example, if we would like to make sure that if allocate in server_b_SUITE (above) crashes, deallocate is skipped, we may declare the sequence:

        sequences() -> [{alloc_and_dealloc,[alloc,dealloc]}].

Let's also assume the suite contains the test case get_resource_status, which is independent of the other two cases. The all function could look like this:

        all() -> [{sequence,alloc_and_dealloc}, get_resource_status].

If alloc succeeds, dealloc is also executed. If alloc fails however, dealloc is not executed but marked as SKIPPED in the html log. get_resource_status will run no matter what happens to the alloc_and_dealloc cases.

Test cases in a sequence will be executed in order until they have all succeeded or until one case fails. If one fails, all following cases in the sequence are skipped. The cases in the sequence that have succeeded up to that point are reported as successful in the log. An arbitrary number of sequence tuples may be specified. Example:

        sequences() -> [{scenarioA, [testA1, testA2]}, 
                        {scenarioB, [testB1, testB2, testB3]}].

        all() -> [test1, 
                  test2, 
                  {sequence,scenarioA}, 
                  test3, 
                  {sequence,scenarioB}, 
                  test4].

Note

It is not possible to execute a test case which is part of a sequence as a regular (stand alone) test case. It is also not possible to use the same test case in multiple sequences. Remember that you can always work around these limitations if necessary by means of help functions common for the test cases in question.