[erlang-questions] Trace-Driven Development

[Ulf Wiger]

On 6 Jun 2012, at 08:02, Michael Turner wrote:

> There are three issues here:
> (1) how honest one should be in a given context,

Indeed, and perhaps also in what *context* that honesty
even applies and can be considered correct.

Erlang tracing may be a "seething pile of pain" from one
perspective, yet such a statement can easily convey the
idea that tracing in Erlang is inferior to tracing in other 
language environments.

Technology constantly evolves, of course, and this 
could eventually become true (a bit like Erlang's error
messages were considered very helpful until python
came along and set a new standard - eventually 
forcing Erlang to improve too).

> This thread dissolved into acrimony between me and Ulf, 
> and I believe largely because seq_trace has been 
> somewhat the victim of a "worst of both
> worlds" scenario, of a kind I can only speculate about 
> since the disambiguating details are behind the 
> corporate veil.

Discussions in a public forum do not need to dissolve
into acrimony just because there is disagreement.
I tried to offer *some* insight into the background of
seq_trace, but you need to respect the fact that I am
not at liberty to discuss Ericsson-internal project details,
*especially* since I no longer work for Ericsson.

But I can share part of the blame. Offering you a public
link mentioning forlopp tracing in AXE was not intended
to mean "look, here is proof of X", but mostly as a kind 
of alibi for myself - since there exists documentation on 
the net mentioning these Ericsson-proprietary aspects,
I consider myself justified in mentioning them, without
violating my confidentiality obligations to my former
employer. I wasn't clear about this intent, and you read
something else into it.

Nor did I ever try to claim that Ericsson 'invented' either
tracing, forlopp tracing or Lamport clocks. I have observed,
though, in other contexts, that some of the early work at 
Ericsson that led, among other things, to Erlang, actually
happened in *parallel* with much of the seminal work
by Dijkstra, Lamport, Hoare et al. Bjarne Däcker, Mike
Williams and others were discussing these things internally
already in the '70s (I have seen some cute stencil reports,
clearly typewriter-written with traditional glue used to
insert pictures, '70s discussions about the importance of
selective message reception, long before Erlang came 
about). The designs of the AXE, laid down in the
early '70s, where to some (probably fairly large) extent 
based on the experiences of building the AKE switch
in the '60s. Surely they were also tracking closely what
happened at e.g. Bell Labs, since they were pushing the 
envelope at the same time. Ericsson was actually a 
bit player until the AXE came to dominate the switch
market, and its designs had a *huge* influence internally.

http://www.ericssonhistory.com/templates/Ericsson/Article.aspx?id=2095&ArticleID=1384&CatID=362&epslanguage=EN

http://books.google.se/books?id=07NmhqkOqwsC&pg=PR14&lpg=PR14#v=onepage&q&f=false (pg 233)

Of course, even though the AXE *control system* was
single-CPU, it was a multi-processor system, and telephony
systems have formed distributed systems, using signaling
protocols, since at least the 60s - see the above book,
page 451, for example. I don't know if they did tracing across
processors or across switches even. I have no documents
describing AXE tracing to that detail, and I have not worked
on the AXE myself (although many of my colleagues at the
time had).

One technology that I long wondered if Ericsson had 
invented was the Specification and Description Language,
SDL. It was so pervasive at Ericsson, and seemed to have 
been used from the beginning of time. On page 267 in the
above book, the origins are traced to Kawashima 1971,
but LM Ericsson was represented in the early standardization
work at the time, and was certainly one of the early adopters,
possibly also shaping parts of the standard.

For this reason, I've assumed the habit of not excluding
the possibility that certain things were 'known' inside
large telecom companies (at least among a select few)
before they were made known to the world, perhaps by
some other institution. But to find out, you typically have 
to sit down with some old guy and talk to them - maybe
they even have some old stencils archived that they
can pull out. You won't find out through normal 
citation searches.

This was all before the Web. Sorting out who informed
whom this many years afterwards is not easy, and when
building products in a proprietary environment, one is 
usually not in a habit of doing so (although one should).
I would not expect industrial programmers in the 90s
to be much helped by mentions of Lamport clocks, however
technically accurate. Many Ericsson programmers at the
time, though, *would* be helped by examples comparing 
to AXE and MD110.

Again, this is just background. Obviously, in the age of 
the Web, Open Source and NoSQL, the documentation needs
to evolve, become more transparent and relevant to the
programmers of today. For this purpose, I think the OTP
documentation process is too formal. Blogs, wikis, etc
are an excellent complement, but community contributions
to the OTP documentation is also a good way forward.

Erlang does straddle two worlds - one that is pretty fanatical
about transparency, and one that is almost exactly the 
opposite. Much of Erlang's documentation evolved to serve
the latter. The discussion about the "special kind of hell" quote
illustrates what tends to happen when perspectives collide.

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com