[erlang-questions] eunit vs. common_test

[David Mercer]

> >   I would write it in common_test as:
> >
> >        fib_test(_) ->
> >                1 = fib(0),
> >                1 = fib(1),
> >                2 = fib(2),
> >                3 = fib(3),
> >                4 = fib(4),
> >                8 = fib(5),
> >                {'EXIT',{function_clause,_}} = catch fib(-1),
> >                2178309 = fib(31).
. . .
> add fib(4) = 1337 (to simulate a bug) in your fib-implementation and
> see how your tests goes.
> 
> Your eunit fib_test_ is a test generator, and ?_assert() returns an
> individual test for your list of tests. Each test will be reported for
> failure/success.

Very true, and I would actually have written these test cases as separate
test cases in common_test:

	fib__0(_) -> 1 = fib(0).
	fib__1(_) -> 1 = fib(1).
	. . .
	fib__7(_) -> 2178309 = fib(31).

Which gets around the problem you describe, but does point out what I think
is the biggest pain in using common_test: every test you make should be
returned by your all/0 function.  So if I were to add an eighth testcase, it
would not be enough for me to write "fib__8(_) -> ...", but I would also
have to remember to add it to the all/0 function so that it would actually
be run.  If I forget to do this, and the test fails, I would be oblivious to
it.  The other downside to common_test is having to name each test, and
since common_test writes out files using the names you choose, they cannot
be arbitrary names describing the purpose of the test, since they have to
form legal filenames.

On the other hand, what I do like about common_test is the easy coverage
analysis and the nice HTML reports (in that order).  I run common_test
frequently as I develop, using it as a unit-testing tool.  I see the
advantage, however, of having tests right next to the code it is testing as
is the case with eunit.  I may experiment a bit to see what sort of output I
get from eunit, and see if the nice coverage reports are available if I were
to use eunit.  If nothing else, the idea of using common_test to run eunit
would get me both the nice coverage analysis and the tests right next to the
code they are testing...

Thanks for the advice.

David

> -----Original Message-----
> From: Christian [mailto:chsu79@REDACTED]
> Sent: Friday, June 12, 2009 12:45 PM
> To: David Mercer
> Cc: erlang-questions
> Subject: Re: [erlang-questions] eunit vs. common_test
> 
> I think that usage of blackbox and whitebox testing is off. But I'm
> not sure how well defined those terms are in testing. This is how I
> use them at work:
> 
> To me blackbox testing is to:
> * be fully aware of the implementation
> * to derive tests cases from the code knowledge
> * to reach into components that are not available to the application user
> 
> And whitebox testing is:
> * to act as a user, preferably to avoid learning implementation details
> * to drive test cases from a functional description/specification of the
> system
> * to use the abstractions and simplifications the application presents
> the user with
> 
> If there is any distiction I would put into eunit vs ct it is that
> eunit is for quick tests, to be run each time you save your buffers or
> at least before you check in your code.
> 
> Comparably ct is more for daily smoke-tests, or very long-running
> duration tests. But as you see for yourself, both tools have some
> overlap and with some stretching you can use ct for unit tests and
> eunit for functional tests.
> 
> By the way,
> 
> add fib(4) = 1337 (to simulate a bug) in your fib-implementation and
> see how your tests goes.
> 
> Your eunit fib_test_ is a test generator, and ?_assert() returns an
> individual test for your list of tests. Each test will be reported for
> failure/success.
> 
> Your ct test fib_test will check that all tests succeed or if at least
> one failed.
> 
> In this case: eunit wins even though all that
> questionmark-leading-underscore macrology looks like high frequency
> perlish-noise.
> 
> On Fri, Jun 12, 2009 at 18:26, David Mercer<dmercer@REDACTED> wrote:
> > Would it be accurate to describe common_test as a blackbox testing tool
> and
> > eunit as a whitebox (or open-box or whatever the opposite of blackbox
> is)?
> >
> > If so, is the following example in the eunit documentation misleading?
>  I
> > usually think of asserts as being in the code rather than separated in
> > separate test functions:
> >
> >        fib_test_() ->
> >           [?_assert(fib(0) =:= 1),
> >                ?_assert(fib(1) =:= 1),
> >                ?_assert(fib(2) =:= 2),
> >                ?_assert(fib(3) =:= 3),
> >                ?_assert(fib(4) =:= 5),
> >                ?_assert(fib(5) =:= 8),
> >                ?_assertException(error, function_clause, fib(-1)),
> >                ?_assert(fib(31) =:= 2178309)
> >           ].
> >
> > This is a blackbox test, and I would write it in common_test as:
> >
> >        fib_test(_) ->
> >                1 = fib(0),
> >                1 = fib(1),
> >                2 = fib(2),
> >                3 = fib(3),
> >                4 = fib(4),
> >                8 = fib(5),
> >                {'EXIT',{function_clause,_}} = catch fib(-1),
> >                2178309 = fib(31).
> >
> > Am I on-track in my understanding, or completely off-base?
> >
> >> -----Original Message-----
> >> From: Christian [mailto:chsu79@REDACTED]
> >> Sent: Friday, June 12, 2009 9:57 AM
> >> To: David Mercer
> >> Cc: erlang-questions
> >> Subject: Re: [erlang-questions] eunit vs. common_test
> >>
> >> I do not feel that they are two overlapping test systems where you
> >> pick one or the other but not both.
> >>
> >> Eunit has tools for assertions and making table driven tests, things
> >> that are common in unit tests. A tool for the developer.
> >>
> >> Common test is more of a manager of test suites, to give them a
> >> configuration and to set up multiple nodes to participate in a test.
> >> Ct is good at giving you test code coverage with relatively little
> >> effort. A tool for someone that manages the test machines and makes
> >> sure the managers and developers get their reports.
> >>
> >> One test suite in ct can very well be to execute your eunit tests.
> >>
> >>
> >>
> >> On Fri, Jun 12, 2009 at 15:46, David Mercer<dmercer@REDACTED> wrote:
> >> > It seems that most of us use eunit for unit testing instead of
> >> common_test.
> >> > I myself use common_test.  What are the advantages eunit has over
> >> > common_test, and should I switch?  Please advise.  Thank-you.
> >> >
> >> >
> >> >
> >> > David Mercer
> >> >
> >> >
> >
> >