[erlang-questions] Trace-Driven Development

[Tim Watson]

On 6 Jun 2012, at 01:40, Henning Diedrich wrote:

> I think from the pyre, this must be salvaged as the actual point I was going for:
> 
> On 6/4/12 6:52 AM, Michael Turner wrote:
>> 
>> I'm now
>> doing unit testing on a module as I develop it further, based on
>> collecting and filtering seq_trace results
> 
> Is this a common approach, who else does this and how? 
> 

I do the 'collect intermediate data for later verification' a fair bit, sometimes using seq_trace, but not always. 

> Using self-written stuff based on seq_trace, or et_collector?
> 

I had been doing self written stuff - common_test configuration that gets 'auto applied' during init_per_suite/init_per_testcase. I have also used the dbg module (and redbug from time to time, after discovering it by accident when using eper to look at node monitoring approaches) to simply log useful trace information during test runs, which can help easily diagnose why a test is failing.

I have thought many times that the (various?) tracing facilities could make for the foundations of a nice testing framework. I am now going to play with et_collector - an application which I'd not paid any attention to TBH - and see where all the different pieces end up.

BTW, if anyone has built, or plans to build, stable and general purpose testing tools based on the trace facilities, I will be *VERY* interested. I've been working on some code instrumentation tooling that allows for comments in production code that get transformed into actual function calls when compiled with -Dtest or whatever. One of the several uses I have for this (apart from introducing artificial time delays and the like) is to make trace calls, so et:trace_me is a good target for these kind of things. I had planned to implement the trace calls with some additional meta-variables so you can capture process/call state easily, for example:

%% NB: DO NOT REMOVE - this *comment* is used to generate code during test runs 
%%
%% BUG: arbitrary delays in shutdown notification can lead to cluster failures - bug #99741
%%
%% @trace [pid, stack]
%% @delay [random, 1000, 1000000, ms]

gen_server:reply(ReplyTo, shutting_down),
....

I have not finished thinking about how the trace facilities can/should be used as a general purpose mechanism to verify behaviour yet. Using 'ordinary' tracing is subject to data not making it to the tracer process IIRC, so seq_trace is presumably the way to go if you want to verify things. Setting trace tokens and the like can be easily done with instrumentation, minimising the need to litter production code with 'test-only' expressions. This could be done with annotations, *special* comments or configuration driven build-time transforms. 

Further thoughts about *this* topic would be most interesting IMHO.

>> <snip>