[erlang-questions] Changing scheduler behaviour to make tests less deterministic

[Chris Newcombe]

>>>is Hans Svensson's thesis (with Thomas Aers)

My apologies for the typo.   It was of course Thomas Arts.

I recall seeing the gen_trace.erl code in the mailing list archive a
long time ago.   Perhaps if Hans or Thomas are reading, they could say
whether a more recent version is available?

Is anyone else using trace-abstraction for testing?   It clearly
worked very well testing gen_leader, and that is a non-trivial
problem.

Chris


On 12/28/06, Chris Newcombe <chris.newcombe@REDACTED> wrote:
> > the BIF bump_reductions(N). You could, in principle, insert
> > conditional macroized calls to this to reschedule
> > earlier
>
> Thanks.  I looked briefly at this, and concluded that short of doint
> some kind of global parse-transform (which I've never done) it would
> be too much work to justify the time.   So I'm holding it in reserve
> incase the other options don't work.
>
>
> > Or you could record the trace of events with judicious
> > use of the trace BIFs and capture/process the faulty
> > ones.
>
> One of the coolest papers on real-world software testing I've ever
> read is Hans Svensson's thesis (with Thomas Aers) on the testing of
> gen_leader: http://www.cs.chalmers.se/~hanssv/doc/lic_thesis.ps.  They
> traced multiple test-runs and post-processed them to 'abstract' the
> traces (e.g. identify loops in linear traces), and essentially
> reverse-engineered partial state-transition diagrams.  They
> then inspected those diagrams and found bugs.
>
> But that technique is really orthogonal to injecting more
> non-determinism into the scheduler.  The two would really complement
> each other I think (by giving more varied traces for a given set of
> tests).
>
>
> > Another option might be to license QuickCheck :-)
>
> We are interested in QuckCheck, but haven't tried it yet.  But unless
> I'm missing something, QuickCheck seems to be fairly orthogonal to the
> problem I am addressing, in the same way as the tracing technique
> above.  QuickCheck will generate lots of test cases (hopefully giving
> good coverage of application/function domain space), but those test
> cases will always run fairly deterministically in the current BEAM VM
> unless they send messages to other nodes, or use +K 2 etc.
>
> I think it would be really powerful to combine all three of these
> mechanisms; QuckCheck to generate lots of tests, varying determinism
> in the scheduler to bring out any concurrency issues, and
> trace-abstraction to help analyse the data.
>
> But for now I'm writing my own tests, and inspecting the results
> visually, so the most immediate benefit is the scheduler
> non-determinism.
>
> regards,
>
> Chris
>