[erlang-questions] Trace-Driven Development

[Michael Turner]

Correction: I wrote that the seq_trace documentation "shows traces
"inspired" by AXE, but without the sequence tokens
that would suggest that AXE implemented Lamport clocks -- IF it ever did."

In fact, the seq_trace documentation does feature traces with the
sequence tokens as generated by Lamport's algorithm.

My point, however, stands: "inspired" is not "virtually identical",
and definitely not in this case. In the actual AXE documentation Ulf
points to

  http://www.scribd.com/doc/83784121/AXE-System-Testing-1-Apz-212

the only counter mentioned for tracing is an "error intensity
counter." Lamport clocks generate timestamps based on *two* counters,
neither of them specific to error reporting.

Ulf describes the customer's requirements for seq_trace as "like AXE
only better". If I may suggest: Ericsson improved on the AXE traces by
adding (perhaps among other things) Lamport clocks. There is ample
evidence, in the citations and direct references in at least two
open-literature publications by Erlang luminaries, that they were
quite aware of Lamport's work. Did they just forget all about it, when
writing seq_trace? It's possible, I suppose. Or perhaps an engineer
employed by the customer suggested the feature, without mentioning
Lamport? Who knows? All I know is: if that's what you're doing, that's
what you should call it.

And speaking of what to call things: I don't think you should still be
calling seq_trace "beta", if (as Ulf says) it originated ca 1997.  I'd
do the interface differently, but the more important thing to me now
is stability and correctness.

-michael turner



On Tue, Jun 5, 2012 at 3:01 PM, Michael Turner
<michael.eugene.turner@REDACTED> wrote:
> On Tue, Jun 5, 2012 at 2:56 AM, Ulf Wiger <ulf@REDACTED> wrote:
>
>> Listen, I tried to politely point out to you that the *requirements*
>> for sequence trace in Erlang grew from Ericsson developers'
>> experience using "forlopp tracing" in AXE.
>
> What you actually wrote:
>
> "Anyway, not noting in the docs that sequence trace mimicks AXE's
> forlopp trace seems forgiveable."
>
> But there's nothing in the AXE documentation you pointed me to that
> shows AXE forlopp trace doing Lamport clocks, which is the relevant
> point.
>
> What you actually wrote:
>
> "... many of the inputs that *actually* informed the implementation
> were either confidential or proprietary enough to be of no interest to
> the people reading the manuals."
>
> Yeah, but there's nothing proprietary or confidential about Lamport
> clocks, described in a publication that's cited *directly* in
> publications by about half-a-dozen Erlang contributors in the 1990s.
> Since that Lamport publication is cited in over 2000 other
> publications, including one on the internals of Mnesia, it's hard to
> see how it can be deemed, in advance, "of no interest to the people
> reading the manuals." *I* read the manual on seq_trace. My reaction? I
> expressed considerable confusion, here:
>
>  http://erlang.org/pipermail/erlang-questions/2011-June/059235.html
>
> and here
>
>  https://groups.google.com/group/erlang-programming/browse_thread/thread/3633ef0d84d63bf3/ef39b31f33011f8c
>
> That was before it finally dawned on me: "Lamport clocks! Why the hell
> don't they say so!? Lots has been written about Lamport clocks, and
> undoubtedly somewhere something has been written more clearly and more
> usefully than what I'm looking in here. Obviously, if I want to get
> the most out of seq_trace's trace tokens, reading the seq_trace
> documentation isn't going to get me much farther. I'll have to go to
> the open literature."
>
>> Much like Erlang itself
>> was created - as Joe has described it - as an effort to "make something
>> like AXE, only better".
>
> Yeah, the same Joe Armstrong who's co-author of a book about Erlang
> whose second edition in 1993 cites Leslie Lamport's paper defining
> Lamport clocks.
>
>> This is also why it is called "sequential tracing" (let's call this an
>> educated guess, as I was not part of the naming discussion,
>> although I *was* in the loop when the requirement came up).
>
> If you want to call everything that seq_trace implements "sequential
> tracing," OK, fine. But if you've got Lamport clocks in that
> implementation, and in the API, for God's sake, call them Lamport
> clocks. Say so in the beginning. Lots of people know what to do with
> these. Not giving those people a way to get their usual handle on them
> means you're leaving talent on the table. Telling them early in the
> documentation would inspire confidence in documentation that's so rife
> with basic copyediting errors as to inspire very little confidence
> indeed.
>
>> The people who asked for the feature, and helped finance the
>> development of it, did not ask for Lamport clocks - they asked for
>> something like "forlopp tracing" in AXE. "Forlopp" is a bastardization
>> of the Swedish word "förlopp", which means "sequence". In the
>> world of AXE, it was a way to associate events to a "transaction",
>> which could be restarted if something went wrong.
>
> Seq_trace is not "the feature" -- it's a collection of features. It
> came out of a milieu in which I've identified half-a-dozen people who,
> if their publications in the 1990s are any indication, know perfectly
> well what Lamport clocks are.
>
>> Sequence tracing was therefore a perfectly logical thing to call
>> it in Erlang, as it is one of the correct ways to translate "forlopp
>> trace" to real English.
>>
>> There was no deceit, sloppiness or arrogance behind it.
>
> I can see that there might be "no deceit," but only by way of
> sloppiness. And the sloppiness is pretty obvious.
>
> As for arrogance, well, when you gesture vaguely in the direction of
> AXE documentation that contains no evidence of any independent
> invention of Lamport clocks, in support of the claim "we invented that
> too", and you do it apparently without actually closely comparing that
> the relevant sections of that document, the seq_trace documentation,
> and what Lamport wrote.... OK, I'll be kind, and just call that "yet
> more sloppiness," not arrogance.
>
>> I have told you that the details of AXE's implementation are
>> proprietary. I provided one link to a document that *was* publicly
>> available, and that mentions "forlopp tracing". Take that as a
>> form of existence proof of "sequence tracing" in AXE.
>
> I see a tracing facility there. I don't see Lamport clocks. How many
> times do I have to say this before you'll actually take a look for
> yourself?
>
>> As for "independent verification", am I to understand that I'm being
>> chided for not revealing secret Ericsson design details in a public
>> forum just so you can independently verify my claims?
>
> There's nothing secret about Lamport clocks. Kenneth Lundin said
> seq_trace implements Lamport clocks, in 2007. Mnesia uses Lamport
> clocks, as reported in 1999. The first book about Erlang (2nd ed.
> 1993) refers to the 1978 paper in which Lamport clocks were defined.
> Where is the excuse for *not* saying that seq_trace implements Lamport
> clocks?
>
> [snip]
>
>> I did agree that the reference to Lamport clocks could be added to
>> the documentation, since it is not without interest. The one thing
>> I don't agree with is that the OTP team deserves infamy for not
>> having done so already.
>
> I don't believe the entire team deserves infamy. I DO believe that
> whoever's responsible for documenting seq_trace has been, at best,
> very lax.
>
> As for any actual motivation to obscure origins, the best I can come
> up with is one that is still very compelling indeed: having a
> grotesquely swollen patent pool, for purposes of intimidation and/or
> deterrence. After all, it's much easier to get a patent attorney to
> write one of those not-very-inventive software patents if you say,
> "well, there really IS no prior art on this one, except our own, of
> course. Which is proprietary." At its extremes, you get disgusted
> engineering teams pulling pranks like the one James Gosling described:
> a draft patent describing the electrical power switch.
>
>  http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=44&f=G&l=50&co1=AND&d=PTXT&s1=%22gosling,+james%22.INNM.&OS=IN/%22gosling,+james%22&RS=IN/%22gosling,+james%22
>
>  http://www.theregister.co.uk/2010/08/17/golsing_on_sun_goofy_patent_contestas/
>
>> When publishing research, it is extremely bad form not to mention
>> prior art. When writing a user guide, one has to consider whether
>> describing details of the implementation actually helps the user,
>> *and* whether those details are something one wants to commit
>> to as part of the interface.
>>
>> I hope you can appreciate the difference:
>
> Oh, I definitely can. When the documentation isn't very good, a
> reference in it to published research behind its API can at least
> encourage confidence.
>
> I found the seq_trace documentation pretty confusing. I pressed on
> anyway, because I didn't like my alternative: other trace facilities
> that are (literally) described in the company's official documentation
> as hellish. Then I saw how seq_trace generates trace tokens, which
> made it sound like some serious thought had been put into seq_trace --
> if only I could figure out the implications of the algorithm. After a
> couple of unanswered questions about this aspect of seq_trace, on this
> list, I finally did figure it out: I was looking at Lamport clocks.
>
> Knowing that seq_trace implements Lamport clocks would have helped me.
> Maybe lots of people never even get as far as I got with seq_trace
> before shrugging and deciding it's something like gs was and is --
> basically a moribund, abandoned line of approach to tracing in
> Erlang/OTP, even if the documentation doesn't come right out and say
> so. (In the case of gs, getting the documentation to come right out
> and say so required prodding from me, after I discovered I was at a
> dead end with it.)
>
> That's a waste. An easily remedied waste.
>
> How long does something have to stay in beta? I'm finding other
> versions of the seq_trace documentation with copyrights going back to
> 1991. If, as you say, the requirements come out of the late 90s,
> that's like FIFTEEN YEARS IN BETA.
>
>> - Kenneth agreeing on a mailing list that Lamport clocks are used
>>  is a service to the community, but not a commitment to stay with
>>  this design choice indefinitely (not that I think there are that many
>>  other good ways to do it).
>
> You mean if I write to seq_trace as a spec that uses a pretty standard
> (and in fact famous) way of generating trace tokens, it might change
> out from under me? May I ask, why? To what? Or would that be inquiring
> of Ericsson's trade secrets?
>
>> - Inserting details about the implementation in the Reference Manual
>>  elevates that information to become part of the interface.
>
> In the case of seq_trace implementing Lamport clocks, that sounds like
> an excellent idea to me, given the vintage of the idea and how, in the
> test of time, that idea has withstood over three decades.
>
>> Another, perhaps more appropriate, place to insert the reference is
>> as a comment in the source code. It isn't there, however.
>
> Gosh, yet another place where you save people some comprehension time
> by just writing four words. I wonder how many more?
>
>> Granted, the interface *as described* pretty much commits to Lamport
>> clocks. It does seem reasonable to mention that, although I'm still
>> not convinced that many will find it that helpful.
>
> Of course not. After all, somebody would have to read all 2000+ papers
> that cite Lamport's paper, before they could be *really* sure.
>
> [snip]
>> Perhaps this thesis can be of interest then?
>>
>> http://www.student.nada.kth.se/~nstahle/staahle_niklas.pdf
>>
>> (Interestingly, the thesis project was run at Ericsson, on real-time
>> 'transaction' or 'sequence' tracing. While it mentions Lamport and
>> a few other techniques, it fails to mention that both the AXE and
>> Erlang - both established Ericsson technologies at the time -
>> already supported this feature. ;-)
>
> Yeah, when you don't identify the wheel by name, making it hard to
> find in a search, people reinvent that wheel. How amazing!
>
> I've also seen a professor's assignment online: he starts with simple
> trace framework in Erlang, and the student is then asked to implement
> Lamport clocks. No mention of seq_trace. I guess it's because that
> professor searched on "Lamport clocks" in the Erlang documentation and
> said, "Hm, that's odd: no Lamport clocks."
>
> Nice job of hiding gold. Next step: cast doubt on whether gold is
> actually all that valuable, when compared to something unstated that
> you might invent one of these days.
>
>>>> The OTP team is known to accept patches to the documentation,
>>>> so please feel free to contribute to a more helpful way to describe
>>>> the tracing support. I'm sure it would be universally appreciated.
>>>
>>> The one time I tried to get a patch into anything in Erlang (something
>>> that was causing a build under FreeBSD to segfault), it didn't make it
>>> into the next release -- even though other people helped get it into
>>> shape for submission. Pardon me for being a little leery of the
>>> process.
>>
>> Patching can be pretty hairy, but note that patching the documentation
>> is much easier than patching the emulator. ;-)
>
> I'm sure it *can* be hairy, but mine was a single-character change, as
> I recall, and it only asked that a C auto array have a dimension of 1
> rather than zero (a write to that array unsurprising caused the stack
> to be trashed on the platform I was trying to build on, which made
> using source-level debugging a non-starter, which made debugging a
> total pain in the ass.)
>
>>> And I suggest that you NOT take the opportunity to add any supposed
>>> "history" about how AXE implemented Lamport clocks until you can
>>> demonstrate that it did. From what I can see in the documentation you
>>> pointed to, AXE did not.
>>
>> Actually, there is some mention of it in the seq_trace docs already:
>
> [snip]
>
> It shows traces "inspired" by AXE, but without the sequence tokens
> that would suggest that AXE implemented Lamport clocks -- IF it ever
> did.
>
>> More Ericsson history than that is hardly needed, ...
>
> I'd say that history is entirely dispensable. I'd even throw out the
> AXE traces, if you could use the space to instead say something about
> how useful the trace tokens can be.
>
>> ... but does
>> somewhat strengthen my claim that these systems were the main
>> source of inspiration.
>
> We're not talking about whether AXE was "the main source of
> inspiration". We're talking about whether the seq_trace documentation
> (especially given how confusing it is to begin with) should mention
> that it implements Lamport clocks, since, after all, (a) it does, and
> (b) lots of people who do concurrent programming know what Lamport
> clocks are already -- including about half-a-dozen in the Erlang group
> (at least if what they cite in their publications is any indication.)
>
> -michael turner