[erlang-questions] Changing scheduler behaviour to make tests less deterministic

[Chris Newcombe]

> the BIF bump_reductions(N). You could, in principle, insert
> conditional macroized calls to this to reschedule
> earlier

Thanks.  I looked briefly at this, and concluded that short of doint
some kind of global parse-transform (which I've never done) it would
be too much work to justify the time.   So I'm holding it in reserve
incase the other options don't work.


> Or you could record the trace of events with judicious
> use of the trace BIFs and capture/process the faulty
> ones.

One of the coolest papers on real-world software testing I've ever
read is Hans Svensson's thesis (with Thomas Aers) on the testing of
gen_leader: http://www.cs.chalmers.se/~hanssv/doc/lic_thesis.ps.  They
traced multiple test-runs and post-processed them to 'abstract' the
traces (e.g. identify loops in linear traces), and essentially
reverse-engineered partial state-transition diagrams.  They
then inspected those diagrams and found bugs.

But that technique is really orthogonal to injecting more
non-determinism into the scheduler.  The two would really complement
each other I think (by giving more varied traces for a given set of
tests).


> Another option might be to license QuickCheck :-)

We are interested in QuckCheck, but haven't tried it yet.  But unless
I'm missing something, QuickCheck seems to be fairly orthogonal to the
problem I am addressing, in the same way as the tracing technique
above.  QuickCheck will generate lots of test cases (hopefully giving
good coverage of application/function domain space), but those test
cases will always run fairly deterministically in the current BEAM VM
unless they send messages to other nodes, or use +K 2 etc.

I think it would be really powerful to combine all three of these
mechanisms; QuckCheck to generate lots of tests, varying determinism
in the scheduler to bring out any concurrency issues, and
trace-abstraction to help analyse the data.

But for now I'm writing my own tests, and inspecting the results
visually, so the most immediate benefit is the scheduler
non-determinism.

regards,

Chris