[erlang-questions] EUnit not failing test cases which timeout

[Ignas Vyšniauskas]

Hi,

if I have a test case which is timing out for some reason, EUnit will
not consider that as a failure, i.e. a module like this:

-module(foobar).
-include_lib("eunit/include/eunit.hrl").

timing_out_test() ->
    timer:sleep(10001).

will produce this:

======================== EUnit ========================
foobar: timing_out_test (module 'foobar')...*timed out*
undefined
=======================================================
  Failed: 0.  Skipped: 0.  Passed: 0.
One or more tests were cancelled.
ERROR: One or more eunit tests failed.

As you can see in the last 3 lines, there is a bunch of information
which contradicts itself. Terrible!

On the other hand, eunit_surefire will generate a report which claims
the test was skipped:

<testsuite tests="1" failures="0" errors="0" skipped="1" time="0.000"
name="module 'foobar'">
  <testcase time="0.000" name="foobar:timing_out_test/0_0"
description="module 'foobar'">
    <skipped type="timeout"/>
    <system-out>
    </system-out>
  </testcase>
</testsuite>

Why is timing out being considered a skipped test case rather than a
failed one? Time-out's happen quite a lot in Erlang, but due to this,
one might not experience the failure. I believe this is wrong and tests
which time-out should be marked as failures.

It would be good to have any feedback on this.

Thanks,
Ignas

P.S. Maybe I should post this to erlang-bugs instead?