[erlang-questions] Testing data with subsets using QuickCheck

[Jesper Louis Andersen]

On Wed, May 10, 2017 at 6:18 PM Krzysztof Jurewicz <
krzysztof.jurewicz@REDACTED> wrote:

> Let’s say that we want to test lists:member/2 using
> QuickCheck/triq/PropEr. We want to ensure that:
>
> For every element and a list such that the element is member of the list,
> lists:member/2 returns true.
>
> Written as a QuickCheck property, this may look like:
>
> prop_member() ->
>     ?FORALL(
>        {List, Member},
>        list_with_member(),
>        lists:member(Member, List)).
>
>
You have a couple of options:

gen_naive() ->
    ?SUCHTHAT({L,M}, {list(nat()), nat()},
      lists:member(M, L)).

is a simple generator which uses ?SUCHTHAT. It just generates a list and a
member and then verifies if the member is indeed a member of the list. Of
course in this case, when we want to test exactly lists:member/2 this is
silly, but in general that is a way to get at the game.

The rule for a ?SUCHTHAT generator is to use it when you know it is fairly
easy to generate something for which the suchthat is true. If the chance of
generating an valid value is small, then you are just slowing down
generation which means you can do fewer test cases.

Another way is to be a bit smarter:

gen_smarter() ->
    ?LET({L, M, R}, {list(nat()), nat(), list(nat())},
         {L ++ [M] ++ R, M}).

Generate a Left and a Right side and an element. Then append them together
to form a list. This should generate any form of list, not just those where
the element is at the beginning. And this should in principle make the test
cover the domain space well.

 In this case I run a little above 80.000 tests in 10 seconds on this
machine for both approaches, so it is fairly unlikely to slow you down
much. But in general, it is more efficient to construct a specimen in a
generator than search for it through ?SUCHTHAT.

Writing your own custom generators is necessary for any real QuickCheck
work. Automated generators can get you a long way, but if you know
something about your problem domain, it is often possible to improve
testing quite a lot by constructing test cases you know are troublesome far
more often than what a typical autogenerated or derived generator can give
you.

For instance: when I wrote tests for the Erlang maps code, I found a lot of
sibling pairs which hash to the same value in the HAMT. This meant I could
test collisions on hashes all the time and this found some sign extension
bugs in the runtime as a result which got fixed before the release of
Erlang version 18.0.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170510/47b8082e/attachment.htm>