[erlang-questions] Erlang and bbcode

[Ulf Wiger]

On 13 Jul 2012, at 13:04, Joe Armstrong wrote:

> Since they have no specification, their can't really be any tests.
> 
> Or worse, in the absence of a specification, the tests *become* the
> specification.

Well, in general, I'd have to take exception to this description ;-),
but in the case of a markup language, pushing through without 
a specification is obviously asking for trouble.

With common commercial development, there is very
seldom a specification in any useful sense of the word. What 
there is, is a customer's more or less vague idea of what they
want, and the reality any implementation of their idea will 
eventually face.

In such situations, the tests *do* become the specification, 
because a specification *cannot* be written! If you try to write
one anyway, it will seriously delay the project, and be obsolete
as soon as it's finished, if it ever had a chance of being correct
(it probably didn't).

In other words, don't blame the designer who creates a test
suite in those cases, rather than brow-beating the poor customer
until they come up with a workable specification. The practice
of using cue cards to write user stories (and the acceptance 
criteria on the back) evolved very much because it was a 
workable way of discussing the requirements with users, 
whether or not they had any engineering or science background.

http://www.allaboutagile.com/user-story-example/

It seems ridiculously vague and limited, but in practice, it actually
helps to catch *some* of the information needed to arrive at some
mutually understood acceptance criteria. This wasn't created 
because people didn't understand what formal specifications,
or thorough testing, looked like - it was created in order to achieve
at least *some* level of improvement with *real* projects and
real customers.

Besides, when did you last see a good specification of how
a Web GUI should behave - or a good way to test it? *

Another example: When I gave an invited talk at NWPT'99 [1] in 
Uppsala, Sweden, a member of the audience (probably a PdD 
student), reacted to my description of the 3-month test period 
before release of the AXD 301, saying "That's ridiculous! If you 
had written it using a language that could be formally verified, 
you wouldn't have to test at all!".

I mumbled something about not knowing if that would have been
at all possible, and Thomas Arts jumped in and offered to argue 
the point with him in the break. It was a very vivid discussion, as I recall.

[1] http://www.it.uu.se/research/publications/reports/1999-008/nwpt99/proceedings/

Obviously at that scale (for all except for that PhD student), there
is no chance of arriving at a coherent formal specification, since
the standards we were supposed to comply with were not 
themselves formally specified**, and many of the customers didn't
necessarily want *exactly* what was in the standard anyway.
Furthermore, many of the non-functional requirements were not 
specified at all, except as a more or less vague list of "goodness
criteria", trying to encapsulate what it means for a system to be 
robust.

What we ended up with was a large pile of informal requirements,
and a huge set of test cases (nearly a million lines of code).
At that point, the test suites *were* indeed the real specification,
albeit an extremely bulky and inaccessible one.

BR,
Ulf W

* One of the best is probably Cucumber (http://cukes.info/), which
is based on cue cards, and relies on quite an ambitious mapping
to the underlying web stack (e.g. Rails). Without such a mapping,
it is less useful, although it still has its sweet spots.

** …although some would disagree. They are certainly specified in 
a bureaucratic way, but that's not the same thing.

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com