[erlang-questions] Trace-Driven Development

Michael Turner <>
Tue Jun 5 09:47:02 CEST 2012

Correction: I wrote that the seq_trace documentation "shows traces
"inspired" by AXE, but without the sequence tokens
that would suggest that AXE implemented Lamport clocks -- IF it ever did."

In fact, the seq_trace documentation does feature traces with the
sequence tokens as generated by Lamport's algorithm.

My point, however, stands: "inspired" is not "virtually identical",
and definitely not in this case. In the actual AXE documentation Ulf
points to


the only counter mentioned for tracing is an "error intensity
counter." Lamport clocks generate timestamps based on *two* counters,
neither of them specific to error reporting.

Ulf describes the customer's requirements for seq_trace as "like AXE
only better". If I may suggest: Ericsson improved on the AXE traces by
adding (perhaps among other things) Lamport clocks. There is ample
evidence, in the citations and direct references in at least two
open-literature publications by Erlang luminaries, that they were
quite aware of Lamport's work. Did they just forget all about it, when
writing seq_trace? It's possible, I suppose. Or perhaps an engineer
employed by the customer suggested the feature, without mentioning
Lamport? Who knows? All I know is: if that's what you're doing, that's
what you should call it.

And speaking of what to call things: I don't think you should still be
calling seq_trace "beta", if (as Ulf says) it originated ca 1997.  I'd
do the interface differently, but the more important thing to me now
is stability and correctness.

-michael turner

On Tue, Jun 5, 2012 at 3:01 PM, Michael Turner
<> wrote:
> On Tue, Jun 5, 2012 at 2:56 AM, Ulf Wiger <> wrote:
>> Listen, I tried to politely point out to you that the *requirements*
>> for sequence trace in Erlang grew from Ericsson developers'
>> experience using "forlopp tracing" in AXE.
> What you actually wrote:
> "Anyway, not noting in the docs that sequence trace mimicks AXE's
> forlopp trace seems forgiveable."
> But there's nothing in the AXE documentation you pointed me to that
> shows AXE forlopp trace doing Lamport clocks, which is the relevant
> point.
> What you actually wrote:
> "... many of the inputs that *actually* informed the implementation
> were either confidential or proprietary enough to be of no interest to
> the people reading the manuals."
> Yeah, but there's nothing proprietary or confidential about Lamport
> clocks, described in a publication that's cited *directly* in
> publications by about half-a-dozen Erlang contributors in the 1990s.
> Since that Lamport publication is cited in over 2000 other
> publications, including one on the internals of Mnesia, it's hard to
> see how it can be deemed, in advance, "of no interest to the people
> reading the manuals." *I* read the manual on seq_trace. My reaction? I
> expressed considerable confusion, here:
>  http://erlang.org/pipermail/erlang-questions/2011-June/059235.html
> and here
>  https://groups.google.com/group/erlang-programming/browse_thread/thread/3633ef0d84d63bf3/ef39b31f33011f8c
> That was before it finally dawned on me: "Lamport clocks! Why the hell
> don't they say so!? Lots has been written about Lamport clocks, and
> undoubtedly somewhere something has been written more clearly and more
> usefully than what I'm looking in here. Obviously, if I want to get
> the most out of seq_trace's trace tokens, reading the seq_trace
> documentation isn't going to get me much farther. I'll have to go to
> the open literature."
>> Much like Erlang itself
>> was created - as Joe has described it - as an effort to "make something
>> like AXE, only better".
> Yeah, the same Joe Armstrong who's co-author of a book about Erlang
> whose second edition in 1993 cites Leslie Lamport's paper defining
> Lamport clocks.
>> This is also why it is called "sequential tracing" (let's call this an
>> educated guess, as I was not part of the naming discussion,
>> although I *was* in the loop when the requirement came up).
> If you want to call everything that seq_trace implements "sequential
> tracing," OK, fine. But if you've got Lamport clocks in that
> implementation, and in the API, for God's sake, call them Lamport
> clocks. Say so in the beginning. Lots of people know what to do with
> these. Not giving those people a way to get their usual handle on them
> means you're leaving talent on the table. Telling them early in the
> documentation would inspire confidence in documentation that's so rife
> with basic copyediting errors as to inspire very little confidence
> indeed.
>> The people who asked for the feature, and helped finance the
>> development of it, did not ask for Lamport clocks - they asked for
>> something like "forlopp tracing" in AXE. "Forlopp" is a bastardization
>> of the Swedish word "förlopp", which means "sequence". In the
>> world of AXE, it was a way to associate events to a "transaction",
>> which could be restarted if something went wrong.
> Seq_trace is not "the feature" -- it's a collection of features. It
> came out of a milieu in which I've identified half-a-dozen people who,
> if their publications in the 1990s are any indication, know perfectly
> well what Lamport clocks are.
>> Sequence tracing was therefore a perfectly logical thing to call
>> it in Erlang, as it is one of the correct ways to translate "forlopp
>> trace" to real English.
>> There was no deceit, sloppiness or arrogance behind it.
> I can see that there might be "no deceit," but only by way of
> sloppiness. And the sloppiness is pretty obvious.
> As for arrogance, well, when you gesture vaguely in the direction of
> AXE documentation that contains no evidence of any independent
> invention of Lamport clocks, in support of the claim "we invented that
> too", and you do it apparently without actually closely comparing that
> the relevant sections of that document, the seq_trace documentation,
> and what Lamport wrote.... OK, I'll be kind, and just call that "yet
> more sloppiness," not arrogance.
>> I have told you that the details of AXE's implementation are
>> proprietary. I provided one link to a document that *was* publicly
>> available, and that mentions "forlopp tracing". Take that as a
>> form of existence proof of "sequence tracing" in AXE.
> I see a tracing facility there. I don't see Lamport clocks. How many
> times do I have to say this before you'll actually take a look for
> yourself?
>> As for "independent verification", am I to understand that I'm being
>> chided for not revealing secret Ericsson design details in a public
>> forum just so you can independently verify my claims?
> There's nothing secret about Lamport clocks. Kenneth Lundin said
> seq_trace implements Lamport clocks, in 2007. Mnesia uses Lamport
> clocks, as reported in 1999. The first book about Erlang (2nd ed.
> 1993) refers to the 1978 paper in which Lamport clocks were defined.
> Where is the excuse for *not* saying that seq_trace implements Lamport
> clocks?
> [snip]
>> I did agree that the reference to Lamport clocks could be added to
>> the documentation, since it is not without interest. The one thing
>> I don't agree with is that the OTP team deserves infamy for not
>> having done so already.
> I don't believe the entire team deserves infamy. I DO believe that
> whoever's responsible for documenting seq_trace has been, at best,
> very lax.
> As for any actual motivation to obscure origins, the best I can come
> up with is one that is still very compelling indeed: having a
> grotesquely swollen patent pool, for purposes of intimidation and/or
> deterrence. After all, it's much easier to get a patent attorney to
> write one of those not-very-inventive software patents if you say,
> "well, there really IS no prior art on this one, except our own, of
> course. Which is proprietary." At its extremes, you get disgusted
> engineering teams pulling pranks like the one James Gosling described:
> a draft patent describing the electrical power switch.
>  http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=44&f=G&l=50&co1=AND&d=PTXT&s1=%22gosling,+james%22.INNM.&OS=IN/%22gosling,+james%22&RS=IN/%22gosling,+james%22
>  http://www.theregister.co.uk/2010/08/17/golsing_on_sun_goofy_patent_contestas/
>> When publishing research, it is extremely bad form not to mention
>> prior art. When writing a user guide, one has to consider whether
>> describing details of the implementation actually helps the user,
>> *and* whether those details are something one wants to commit
>> to as part of the interface.
>> I hope you can appreciate the difference:
> Oh, I definitely can. When the documentation isn't very good, a
> reference in it to published research behind its API can at least
> encourage confidence.
> I found the seq_trace documentation pretty confusing. I pressed on
> anyway, because I didn't like my alternative: other trace facilities
> that are (literally) described in the company's official documentation
> as hellish. Then I saw how seq_trace generates trace tokens, which
> made it sound like some serious thought had been put into seq_trace --
> if only I could figure out the implications of the algorithm. After a
> couple of unanswered questions about this aspect of seq_trace, on this
> list, I finally did figure it out: I was looking at Lamport clocks.
> Knowing that seq_trace implements Lamport clocks would have helped me.
> Maybe lots of people never even get as far as I got with seq_trace
> before shrugging and deciding it's something like gs was and is --
> basically a moribund, abandoned line of approach to tracing in
> Erlang/OTP, even if the documentation doesn't come right out and say
> so. (In the case of gs, getting the documentation to come right out
> and say so required prodding from me, after I discovered I was at a
> dead end with it.)
> That's a waste. An easily remedied waste.
> How long does something have to stay in beta? I'm finding other
> versions of the seq_trace documentation with copyrights going back to
> 1991. If, as you say, the requirements come out of the late 90s,
> that's like FIFTEEN YEARS IN BETA.
>> - Kenneth agreeing on a mailing list that Lamport clocks are used
>>  is a service to the community, but not a commitment to stay with
>>  this design choice indefinitely (not that I think there are that many
>>  other good ways to do it).
> You mean if I write to seq_trace as a spec that uses a pretty standard
> (and in fact famous) way of generating trace tokens, it might change
> out from under me? May I ask, why? To what? Or would that be inquiring
> of Ericsson's trade secrets?
>> - Inserting details about the implementation in the Reference Manual
>>  elevates that information to become part of the interface.
> In the case of seq_trace implementing Lamport clocks, that sounds like
> an excellent idea to me, given the vintage of the idea and how, in the
> test of time, that idea has withstood over three decades.
>> Another, perhaps more appropriate, place to insert the reference is
>> as a comment in the source code. It isn't there, however.
> Gosh, yet another place where you save people some comprehension time
> by just writing four words. I wonder how many more?
>> Granted, the interface *as described* pretty much commits to Lamport
>> clocks. It does seem reasonable to mention that, although I'm still
>> not convinced that many will find it that helpful.
> Of course not. After all, somebody would have to read all 2000+ papers
> that cite Lamport's paper, before they could be *really* sure.
> [snip]
>> Perhaps this thesis can be of interest then?
>> http://www.student.nada.kth.se/~nstahle/staahle_niklas.pdf
>> (Interestingly, the thesis project was run at Ericsson, on real-time
>> 'transaction' or 'sequence' tracing. While it mentions Lamport and
>> a few other techniques, it fails to mention that both the AXE and
>> Erlang - both established Ericsson technologies at the time -
>> already supported this feature. ;-)
> Yeah, when you don't identify the wheel by name, making it hard to
> find in a search, people reinvent that wheel. How amazing!
> I've also seen a professor's assignment online: he starts with simple
> trace framework in Erlang, and the student is then asked to implement
> Lamport clocks. No mention of seq_trace. I guess it's because that
> professor searched on "Lamport clocks" in the Erlang documentation and
> said, "Hm, that's odd: no Lamport clocks."
> Nice job of hiding gold. Next step: cast doubt on whether gold is
> actually all that valuable, when compared to something unstated that
> you might invent one of these days.
>>>> The OTP team is known to accept patches to the documentation,
>>>> so please feel free to contribute to a more helpful way to describe
>>>> the tracing support. I'm sure it would be universally appreciated.
>>> The one time I tried to get a patch into anything in Erlang (something
>>> that was causing a build under FreeBSD to segfault), it didn't make it
>>> into the next release -- even though other people helped get it into
>>> shape for submission. Pardon me for being a little leery of the
>>> process.
>> Patching can be pretty hairy, but note that patching the documentation
>> is much easier than patching the emulator. ;-)
> I'm sure it *can* be hairy, but mine was a single-character change, as
> I recall, and it only asked that a C auto array have a dimension of 1
> rather than zero (a write to that array unsurprising caused the stack
> to be trashed on the platform I was trying to build on, which made
> using source-level debugging a non-starter, which made debugging a
> total pain in the ass.)
>>> And I suggest that you NOT take the opportunity to add any supposed
>>> "history" about how AXE implemented Lamport clocks until you can
>>> demonstrate that it did. From what I can see in the documentation you
>>> pointed to, AXE did not.
>> Actually, there is some mention of it in the seq_trace docs already:
> [snip]
> It shows traces "inspired" by AXE, but without the sequence tokens
> that would suggest that AXE implemented Lamport clocks -- IF it ever
> did.
>> More Ericsson history than that is hardly needed, ...
> I'd say that history is entirely dispensable. I'd even throw out the
> AXE traces, if you could use the space to instead say something about
> how useful the trace tokens can be.
>> ... but does
>> somewhat strengthen my claim that these systems were the main
>> source of inspiration.
> We're not talking about whether AXE was "the main source of
> inspiration". We're talking about whether the seq_trace documentation
> (especially given how confusing it is to begin with) should mention
> that it implements Lamport clocks, since, after all, (a) it does, and
> (b) lots of people who do concurrent programming know what Lamport
> clocks are already -- including about half-a-dozen in the Erlang group
> (at least if what they cite in their publications is any indication.)
> -michael turner

More information about the erlang-questions mailing list