[erlang-questions] Common test issues - Possible bug?

[Mikkel Jensen]

Hi Lukas

First of all, thanks for your response!

I have tried using both run_test and erlang shell to start the tests, both
gives me error i described, although the erlang shell usually seems like its
able to execute more repeated tests than the run_test command.

This is the first test case i'm building with Common Test - I have not made
any other test cases, so i have not experienced this in any other context.
Only when using ct_ssh.

I have been trying to narrow down the test case to find specificly what is
causing the errors, but i have been unable to. However the kinds of tests im
running are not too advanced.
I have made a test-case which imitates the behavior of my "real" test case,
just instead of using cnee for imitating user input, the test subject is a
python script which can be controlled directly from the console. In this
python-test, i have not been able to get the error, not even when repeating
the test 500 times or more.

I hope you can find the time to go through my not-so-minimal test case. As
you can probably see from the code, i am not the most experienced erlang
programmer, and the lack of websites/blogs describing Common Test best
practices and examples makes it very hard to know what i'm doing wrong. The
user guide is great, but can sometimes be a bit hard to understand.

Real test case: http://ideone.com/v6vvu
Python test case: http://ideone.com/GcxR2

%%% CODE FOR "REAL" TEST CASE %%%
%%%%%%%%%%%%%%%%%%%%%%%%

-module(migration_SUITE).
-compile(export_all).
-include_lib("common_test/include/ct.hrl").

% Require the following tags before executing the suite.
% The tags are provided by a config file
suite() ->
    [{require, hidenets},
     {require, demo},
     {require, node1},
     {require, node2},
     {require, testserver}].

init_per_suite(Config) ->
% Start ssh connections
{ok, Hidenets} = ct_ssh:connect(hidenets),
 {ok, Demo} = ct_ssh:connect(demo),
{ok, Node1} = ct_ssh:connect(node1),
{ok, Node2} = ct_ssh:connect(node2),
 {ok, Testserver} = ct_ssh:connect(testserver),

% Append handles to config
 % OBS: This is not actually used, because of the disconnect issue
[{ssh_handles,{Hidenets,Demo,Node1,Node2,Testserver}} | Config].
 end_per_suite(Config) ->
% Disconnect from all handles
Handles = erlang:tuple_to_list(?config(ssh_handles, Config)),
 lists:foreach(fun(Handle) -> ct_ssh:disconnect(Handle) end,
Handles).

init_per_group(_GroupName, Config) ->
% Get ssh handles from config
% {Hidenets, Demo, Node1, Node2, Testserver} = ?config(ssh_handles, Config),

% Start context agent on hidenets (simple console application)
runcmd(hidenets, "./ContextAgentLinux/ContextAgent"),

% Start GUI macro playback
        runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee.xns"),
        runcmd(demo, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee.xns"),
        runcmd(node1, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee.xns"),
        runcmd(node2, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee.xns"),
Config.


end_per_group(_GroupName, Config) ->
% Get ssh handles from config
% {Hidenets, Demo, Node1, Node2, Testserver} = ?config(ssh_handles, Config),

% Stop GUY macro playback
runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee-end.xns"),
        runcmd(demo, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee-end.xns"),
        runcmd(node1, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee-end.xns"),
        runcmd(node2, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee-end.xns"),

% Stop context agent on hidenets
 runcmd(hidenets, "pkill Context"),
ok.

% Initialize the testcase
init_per_testcase(_TestCase, Config) ->
Config.

% Execute the following after a test case is finished
end_per_testcase(_TestCase, Config) ->
 ok.

groups() ->
[{migrateTest, [sequence], [migCase1, migCase2]}].

all() ->
% Return a list of all test cases in the suite.
[migrateTest].

migCase1() ->
[].

migCase2() ->
[].

migCase1(Config) ->
%Hidenets = element(1, ?config(ssh_handles,Config)),

% Arbitrary delay to ensure eclipse + CA is started up and running
 ok = timer:sleep(5000),

% Trigger
runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee-triggerzero.xns"),
 ok = timer:sleep(5000),
ok.

migCase2(Config) ->
 % Hidenets = element(1, ?config(ssh_handles,Config)),
% Arbitrary delay to ensure eclipse + CA is started up and running
                ok = timer:sleep(5000),

% Trigger
runcmd(hidenets, "export DISPLAY=:0.0 && cnee --replay -f
~/ct/xnee/xnee-triggerone.xns"),
 ok = timer:sleep(5000),
ok.

% Workaround for disconnect issue
runcmd(HandleCfg, Cmd) ->
case ct_ssh:connect(HandleCfg) of
{ok, Handle} ->
 case ct_ssh:exec(Handle, Cmd, 3000) of
{ok, Data} -> ok;
{error, _} -> runcmd(HandleCfg, Cmd);
 {_, _} -> ok
end;
{error, _} ->
 runcmd(HandleCfg, Cmd);
_ ->
runcmd(HandleCfg, Cmd)
 end.

%%% CODE FOR PYTHON TEST %%%
%%%%%%%%%%%%%%%%%%%%%%

-module(python_migration_SUITE).
-compile(export_all).
-include_lib("common_test/include/ct.hrl").

suite() ->
% Make SSH and Python settings from config files, sshconfig.cfg
 % and pyconfig.cfg, available in the suite
    [{require, server},
     {require, client1},
     {require, client2},
     {require, serverSettings},
     {require, client1Settings},
     {require, client2Settings}].

init_per_suite(Config) ->
    % Start ssh connection to server, client1 and client2
    {ok, Server}  = ct_ssh:connect(server),
    {ok, Client1} = ct_ssh:connect(client1),
{ok, Client2} = ct_ssh:connect(client2),
 % Append handles to config variable
    [{ssh_handles,{Server, Client1, Client2}} | Config].


end_per_suite(Config) ->
% Get ssh handles from config and make them a traversable list
    Handles = erlang:tuple_to_list(?config(ssh_handles, Config)),

    % Disconnect any active ssh handle
lists:foreach(fun(Handle) -> ct_ssh:disconnect(Handle) end,
  Handles).

init_per_group(_GroupName, Config) ->
    % Get ssh handles from config
{Server, Client1, Client2} = ?config(ssh_handles, Config),

    % Start python server on server and log stdout
ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, serverpy})
 ++ " -i " ++ ct:get_config({serverSettings, serverip})
++ " -p " ++ ct:get_config({serverSettings, serverport})
 ++ " >> " ++ ct:get_config({serverSettings, serverlog}),
2000),

    % Start python client on client1 and log stdout
ct_ssh:exec(Client1, "python " ++ ct:get_config({client1Settings,
client1py})
 ++ " -i " ++ ct:get_config({client1Settings, client1ip})
++ " -p " ++ ct:get_config({client1Settings, client1port})
    ++ " > " ++ ct:get_config({client1Settings, client1log}),
2000),

    % Start python client on client2 and log stdout
ct_ssh:exec(Client2, "python " ++ ct:get_config({client2Settings,
client2py})
 ++ " -i " ++ ct:get_config({client2Settings, client2ip})
++ " -p " ++ ct:get_config({client2Settings, client2port})
    ++ " > " ++ ct:get_config({client2Settings, client2log}),
2000),
    Config.

end_per_group(_GroupName, Config) ->
    % Get ssh handles from config
{Server, Client1, Client2} = ?config(ssh_handles, Config),

    % End python server on server
ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, commandpy})
++ " -x"),

    % End python client on client1
ct_ssh:exec(Client1, "pkill -f 'client.py'"),

    % End python client on client2
 ct_ssh:exec(Client2, "pkill -f 'client.py'"),
    ok.

init_per_testcase(_TestCase, Config) ->
    Config.

end_per_testcase(_TestCase, _Config) ->
    ok.

groups() ->
    % Define sequential group migrateC1, then migrate2
    [{migrateTest, [sequence], [migrateC1, migrateC2]}].

all() ->
% Run the whole group when the suite is run
    [migrateTest].

migrateC1() ->
 % Do nothing in this testcase info function
    [].

migrateC2() ->
% Do nothing in this testcase info function
    [].

migrateC1(Config) ->
    % Get server handle as it is the only one used in the testcase
Server = element(1, ?config(ssh_handles,Config)),
 % Initial delay 2 seconds to assure that the server has been started
properly
 ok = timer:sleep(2000),

    % Redirect server stream to client1
ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings, commandpy})
 ++ " -i " ++ ct:get_config({client1Settings, client1ip})
++ " -p " ++ ct:get_config({client1Settings, client1port})),

    % Delay 7 seconds
ok = timer:sleep(7000),
    ok.

migrateC2(Config) ->
    % Get server handle as it is the only one used in the testcase
Server = element(1, ?config(ssh_handles,Config)),

    % Redirect server stream to client2
 ct_ssh:exec(Server, "python " ++ ct:get_config({serverSettings,
commandpy})
++ " -i " ++ ct:get_config({client2Settings, client2ip})
 ++ " -p " ++ ct:get_config({client2Settings, client2port})),

    % Delay 7 seconds
ok = timer:sleep(7000),
    ok.





2010/9/27 Lukas Larsson <lukas@REDACTED>

> Hi Mikkel!
>
> What are you using to run your tests? The run_test command, or are you
> starting them from an erlang shell?
>
> Also have you seen this problem when not using ct_ssh? or is it only
> when you are doing tests with ssh calls in them?
>
> If you could provide me with a minimal test suite which reproduces the
> error you are seeing that would be great!
>
> Regards,
> Lukas, Erlang/OTP, Ericsson AB.
>
> On Fri, Sep 24, 2010 at 6:40 PM, Mikkel Jensen <mikkelgj@REDACTED>
> wrote:
>         Hi list!
>
>        I have been working on setting up a test case using Erlang
>        Common Test. The
>        test is pretty simple, it consists of a sequence of calls to
>        ct_ssh:exec().
>
>        Running the test once works. Running the test a few times (using
>        repeat),
>        usually works. But if i let the test suite repeat a large number
>        of times,
>        it fails after successfully running the test multiple times (In
>        my last
>        attempt it completed 42 runs, no joke, but it seems like the
>        repeated runs
>        can stop at any time).
>
>        The error i'm getting is:
>        It is not possible to install CT while running in interactive
>        mode.
>        To exit this mode, run ct:stop_interactive().
>        To enter the interactive mode again, run ct:start_interactive()
>
>        Does anyone know how i can avoid this, allowing me to run a
>        large number of
>        repeated tests?
>
>        On a sidenote: I have noticed in the logs that ct_ssh will
>        sometimes have
>        lost its connection when trying to call ct_ssh:exec(), and thus
>        will attempt
>        to reconnect. This attempt will, however, always fail instantly.
>        A
>        workaround i have made is to manually connect on each call i
>        make to
>        ct_ssh:exec(). This seems less than ideal, is there a better
>        way?
>
>        Kind regards and thanks in advance,
>        Mikkel
>
>
>
>