testing through trace analysis

[Ulf Wiger]

On 12 Oct 1999, Luke Gorrie wrote:

luke>Hi all,
luke>
luke>I've been playing with a novel (to me) way of testing programs by
luke>tracing their actions while they run, and then at the end
luke>checking that the sequence of events is valid. Like some stuff I
luke>read about in CSP. I did this by hand because it was fairly
luke>convenient to send special messages to a logging process all
luke>within the unit test module.
luke>
luke>Now I've noticed how powerful the trace/3 BIF is for this sort of
luke>thing and I'm really curious to know how people use it. I'm
luke>envisioning a testing framework that uses a little language that
luke>can be used to say what order(s) a set of events (sending
luke>messages, calls to certain modules, etc) can occur - the set of
luke>all possible correct traces - and then have it automatically and
luke>painlessly validated.

We use the trace BIF a lot, but not for esotheric stuff like event
validation.

Personally, I find the trace BIF very useful for debugging of timing-
sensitive sequences in an embedded system. Usually, I enter the trace()
calls from the shell prompt and then use the shell command flush() to
display them (in other words, the shell process gets to collect trace
messages in its message queue, until I'm ready to see them.)

Other uses are:

1. Performance profiling, where we use trace messages with 'proc', 'gc' and
'timestamp' flags to see how much CPU time is spent performing a certain
task.

2. Event-triggered trace, where we can specify how many trace messages
before and after a certain event to display, as well as which event should
trigger the trace presentation (e.g. 10 trace msgs before and 10 after the
reception of message M.)

3. System trace, where we turn on message trace on all processes in the
system and dump them all to disk, in order to figure out what is really
happening when the system behaves strangely.


luke>I've also been generally trying to imagine what the testing
luke>frameworks in use look like, and thinking about things like
luke>having any failed unit test automatically restarted in the
luke>debugger with a breakpoint set at the start, etc. If anyone wants
luke>to satisfy my curiousity as to what testing support works well
luke>I'd be very grateful. :-)

We haven't really invested much energy trying to build generic tools, as we
(or more specifically our profiling guru Kurt) feel that it is just as easy
to write a specific analysis program from scratch when needed.

So much for code reuse...  (:

/Uffe
Ulf Wiger, Chief Designer AXD 301
Ericsson Telecom AB                          tfn: +46  8 719 81 95
Varuvägen 9, Älvsjö                          mob: +46 70 519 81 95
S-126 25 Stockholm, Sweden                   fax: +46  8 719 43 44