[erlang-questions] Common Test repeat_until and exit status

[Loïc Hoguin]

Hello,

When I use the test group property repeat_until_all_ok I still get an 
error status from ct_run (exit status 1) because the tests that failed 
and were repeated are still counted as errors.

For example:



Testing ninenines.gun.gun_SUITE: Starting test (with repeated test cases)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -
gun_SUITE:shutdown_reason failed on line 399
Reason: timeout
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -
gun_SUITE:retry_1 failed on line 324
Reason: timeout
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -

Testing ninenines.gun.gun_SUITE: *** FAILED test case 17 ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -
gun_SUITE:retry_fun failed on line 343
Reason: shutdown_too_late
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -

Testing ninenines.gun.gun_SUITE: *** FAILED test case 18 ***
Testing ninenines.gun.gun_SUITE: *** FAILED test case 21 ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -
gun_SUITE:retry_1 failed on line 324
Reason: timeout
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
- - - -

Testing ninenines.gun.gun_SUITE: *** FAILED test case 46 ***
Testing ninenines.gun.gun_SUITE: Stopping repeat operation 
{repeat_until_all_ok,98}
Testing ninenines.gun.gun_SUITE: TEST COMPLETE, 83 ok, 4 failed of 87 
test cases
[...]
make[1]: *** [erlang.mk:6452: ct] Error 1



Is there a way to repeat flaky tests without Common Test counting the 
test failures as errors? In the example above I want Common Test 
(ct_run) to return the exit status 0 because the tests were OK after a 
few tries.

This would be very useful to quickly get tests green on environments 
that are more likely to have timing issues (in cases where it's the test 
that is the problem, not the application). For example on slower 
environments, or on Windows/FreeBSD where the network stack doesn't 
always behave the same as Linux. I spend far too much time on fixing 
timing issues in tests.

Cheers,

-- 
Loïc Hoguin
https://ninenines.eu