[erlang-questions] common test, test_server and ct_hooks

[Peter Andersson]

Hi Tim,

When a test case times out, the test server control process kills the
test case process, on which the Config data is known (it's last
returned by init_per_testcase/2 on the same hanging process). The
result is that the latest Config value is unknown when the call to the
post_end_per_testcase hook function takes place. This is not a bug as
such (and OTP-9594 is a different issue), but it's a limitation for
sure. There are 2 possible solutions to this problem:

1. You keep saving the last known Config data (or at least your
relevant subset of it) in the hook state. If the test case times out,
you can restore the data from there. No need to use an ets table. You
will be able to restore the Config data that init_per_testcase/2 was
called with, but not the final value that was passed to the test case
function.

2. Common Test gets updated so that the control process saves the last
known Config for a test case so that it can be restored and passed to
the post_end_per_testcase hook call. The result will be the same as #1
above, but it will make it easier to implement the CTH callback
module. It might also be possible to have the test case process send
the Config data to the control process after init_per_testcase, so
that the very latest Config can be saved and restored. Implementation
of this can be somewhat tricky because of parallel test case groups -
but certainly not impossible! ;-)

I'd be very happy (well, sort of) to implement #2 above. Will #1 be an
ok workaround for you until #2 is in place?

I know it's usually not appropriate to be talking about doing #1 and
#2, but in this context you know what I mean! :-)

  /Peter

Ericsson AB, Erlang/OTP

2012/8/13, Tim Watson <watson.timothy@REDACTED>:
> I'm currently using the ct_hooks mechanism to do some automatic
> setup/teardown work between suites, groups and test cases. It works great,
> but I've discovered that under some circumstances, stashing framework
> specific information in the Config variable doesn't work, because when a
> test case fails (specifically with a time trap timeout failure) then the
> configuration set seems to disappear before it arrives in the hook's
> post_end_per_testcase handler. I've tracked the reason down to 1153 of
> test_server:
>
>     %%! This is a temporary fix that keeps Test Server alive during
>     %%! execution of a parallel test case group, when sometimes
>     %%! this clause gets called with EndConf == undefined. See OTP-9594
>     %%! for more info.
>     EndConf1 = if EndConf == undefined ->
> 		       [{tc_status,{failed,{Mod,end_per_testcase,Why}}}];
> 		  true ->
> 		       EndConf
> 	       end,
>
> Where can I go and look at OTP-9594, as I can't understand why my test case
> configuration has disappeared and without this, I'll need to start storing
> the relevant information in an ets table (or equivalent) in order to work
> around this issue.
>
> Cheers,
> Tim
>
>
>