[erlang-questions] Trace-Driven Development

[Michael Turner]

On Tue, Jun 5, 2012 at 2:56 AM, Ulf Wiger <ulf@REDACTED> wrote:

> Listen, I tried to politely point out to you that the *requirements*
> for sequence trace in Erlang grew from Ericsson developers'
> experience using "forlopp tracing" in AXE.

What you actually wrote:

"Anyway, not noting in the docs that sequence trace mimicks AXE's
forlopp trace seems forgiveable."

But there's nothing in the AXE documentation you pointed me to that
shows AXE forlopp trace doing Lamport clocks, which is the relevant
point.

What you actually wrote:

"... many of the inputs that *actually* informed the implementation
were either confidential or proprietary enough to be of no interest to
the people reading the manuals."

Yeah, but there's nothing proprietary or confidential about Lamport
clocks, described in a publication that's cited *directly* in
publications by about half-a-dozen Erlang contributors in the 1990s.
Since that Lamport publication is cited in over 2000 other
publications, including one on the internals of Mnesia, it's hard to
see how it can be deemed, in advance, "of no interest to the people
reading the manuals." *I* read the manual on seq_trace. My reaction? I
expressed considerable confusion, here:

  http://erlang.org/pipermail/erlang-questions/2011-June/059235.html

and here

  https://groups.google.com/group/erlang-programming/browse_thread/thread/3633ef0d84d63bf3/ef39b31f33011f8c

That was before it finally dawned on me: "Lamport clocks! Why the hell
don't they say so!? Lots has been written about Lamport clocks, and
undoubtedly somewhere something has been written more clearly and more
usefully than what I'm looking in here. Obviously, if I want to get
the most out of seq_trace's trace tokens, reading the seq_trace
documentation isn't going to get me much farther. I'll have to go to
the open literature."

> Much like Erlang itself
> was created - as Joe has described it - as an effort to "make something
> like AXE, only better".

Yeah, the same Joe Armstrong who's co-author of a book about Erlang
whose second edition in 1993 cites Leslie Lamport's paper defining
Lamport clocks.

> This is also why it is called "sequential tracing" (let's call this an
> educated guess, as I was not part of the naming discussion,
> although I *was* in the loop when the requirement came up).

If you want to call everything that seq_trace implements "sequential
tracing," OK, fine. But if you've got Lamport clocks in that
implementation, and in the API, for God's sake, call them Lamport
clocks. Say so in the beginning. Lots of people know what to do with
these. Not giving those people a way to get their usual handle on them
means you're leaving talent on the table. Telling them early in the
documentation would inspire confidence in documentation that's so rife
with basic copyediting errors as to inspire very little confidence
indeed.

> The people who asked for the feature, and helped finance the
> development of it, did not ask for Lamport clocks - they asked for
> something like "forlopp tracing" in AXE. "Forlopp" is a bastardization
> of the Swedish word "förlopp", which means "sequence". In the
> world of AXE, it was a way to associate events to a "transaction",
> which could be restarted if something went wrong.

Seq_trace is not "the feature" -- it's a collection of features. It
came out of a milieu in which I've identified half-a-dozen people who,
if their publications in the 1990s are any indication, know perfectly
well what Lamport clocks are.

> Sequence tracing was therefore a perfectly logical thing to call
> it in Erlang, as it is one of the correct ways to translate "forlopp
> trace" to real English.
>
> There was no deceit, sloppiness or arrogance behind it.

I can see that there might be "no deceit," but only by way of
sloppiness. And the sloppiness is pretty obvious.

As for arrogance, well, when you gesture vaguely in the direction of
AXE documentation that contains no evidence of any independent
invention of Lamport clocks, in support of the claim "we invented that
too", and you do it apparently without actually closely comparing that
the relevant sections of that document, the seq_trace documentation,
and what Lamport wrote.... OK, I'll be kind, and just call that "yet
more sloppiness," not arrogance.

> I have told you that the details of AXE's implementation are
> proprietary. I provided one link to a document that *was* publicly
> available, and that mentions "forlopp tracing". Take that as a
> form of existence proof of "sequence tracing" in AXE.

I see a tracing facility there. I don't see Lamport clocks. How many
times do I have to say this before you'll actually take a look for
yourself?

> As for "independent verification", am I to understand that I'm being
> chided for not revealing secret Ericsson design details in a public
> forum just so you can independently verify my claims?

There's nothing secret about Lamport clocks. Kenneth Lundin said
seq_trace implements Lamport clocks, in 2007. Mnesia uses Lamport
clocks, as reported in 1999. The first book about Erlang (2nd ed.
1993) refers to the 1978 paper in which Lamport clocks were defined.
Where is the excuse for *not* saying that seq_trace implements Lamport
clocks?

[snip]

> I did agree that the reference to Lamport clocks could be added to
> the documentation, since it is not without interest. The one thing
> I don't agree with is that the OTP team deserves infamy for not
> having done so already.

I don't believe the entire team deserves infamy. I DO believe that
whoever's responsible for documenting seq_trace has been, at best,
very lax.

As for any actual motivation to obscure origins, the best I can come
up with is one that is still very compelling indeed: having a
grotesquely swollen patent pool, for purposes of intimidation and/or
deterrence. After all, it's much easier to get a patent attorney to
write one of those not-very-inventive software patents if you say,
"well, there really IS no prior art on this one, except our own, of
course. Which is proprietary." At its extremes, you get disgusted
engineering teams pulling pranks like the one James Gosling described:
a draft patent describing the electrical power switch.

  http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=44&f=G&l=50&co1=AND&d=PTXT&s1=%22gosling,+james%22.INNM.&OS=IN/%22gosling,+james%22&RS=IN/%22gosling,+james%22

  http://www.theregister.co.uk/2010/08/17/golsing_on_sun_goofy_patent_contestas/

> When publishing research, it is extremely bad form not to mention
> prior art. When writing a user guide, one has to consider whether
> describing details of the implementation actually helps the user,
> *and* whether those details are something one wants to commit
> to as part of the interface.
>
> I hope you can appreciate the difference:

Oh, I definitely can. When the documentation isn't very good, a
reference in it to published research behind its API can at least
encourage confidence.

I found the seq_trace documentation pretty confusing. I pressed on
anyway, because I didn't like my alternative: other trace facilities
that are (literally) described in the company's official documentation
as hellish. Then I saw how seq_trace generates trace tokens, which
made it sound like some serious thought had been put into seq_trace --
if only I could figure out the implications of the algorithm. After a
couple of unanswered questions about this aspect of seq_trace, on this
list, I finally did figure it out: I was looking at Lamport clocks.

Knowing that seq_trace implements Lamport clocks would have helped me.
Maybe lots of people never even get as far as I got with seq_trace
before shrugging and deciding it's something like gs was and is --
basically a moribund, abandoned line of approach to tracing in
Erlang/OTP, even if the documentation doesn't come right out and say
so. (In the case of gs, getting the documentation to come right out
and say so required prodding from me, after I discovered I was at a
dead end with it.)

That's a waste. An easily remedied waste.

How long does something have to stay in beta? I'm finding other
versions of the seq_trace documentation with copyrights going back to
1991. If, as you say, the requirements come out of the late 90s,
that's like FIFTEEN YEARS IN BETA.

> - Kenneth agreeing on a mailing list that Lamport clocks are used
>  is a service to the community, but not a commitment to stay with
>  this design choice indefinitely (not that I think there are that many
>  other good ways to do it).

You mean if I write to seq_trace as a spec that uses a pretty standard
(and in fact famous) way of generating trace tokens, it might change
out from under me? May I ask, why? To what? Or would that be inquiring
of Ericsson's trade secrets?

> - Inserting details about the implementation in the Reference Manual
>  elevates that information to become part of the interface.

In the case of seq_trace implementing Lamport clocks, that sounds like
an excellent idea to me, given the vintage of the idea and how, in the
test of time, that idea has withstood over three decades.

> Another, perhaps more appropriate, place to insert the reference is
> as a comment in the source code. It isn't there, however.

Gosh, yet another place where you save people some comprehension time
by just writing four words. I wonder how many more?

> Granted, the interface *as described* pretty much commits to Lamport
> clocks. It does seem reasonable to mention that, although I'm still
> not convinced that many will find it that helpful.

Of course not. After all, somebody would have to read all 2000+ papers
that cite Lamport's paper, before they could be *really* sure.

[snip]
> Perhaps this thesis can be of interest then?
>
> http://www.student.nada.kth.se/~nstahle/staahle_niklas.pdf
>
> (Interestingly, the thesis project was run at Ericsson, on real-time
> 'transaction' or 'sequence' tracing. While it mentions Lamport and
> a few other techniques, it fails to mention that both the AXE and
> Erlang - both established Ericsson technologies at the time -
> already supported this feature. ;-)

Yeah, when you don't identify the wheel by name, making it hard to
find in a search, people reinvent that wheel. How amazing!

I've also seen a professor's assignment online: he starts with simple
trace framework in Erlang, and the student is then asked to implement
Lamport clocks. No mention of seq_trace. I guess it's because that
professor searched on "Lamport clocks" in the Erlang documentation and
said, "Hm, that's odd: no Lamport clocks."

Nice job of hiding gold. Next step: cast doubt on whether gold is
actually all that valuable, when compared to something unstated that
you might invent one of these days.

>>> The OTP team is known to accept patches to the documentation,
>>> so please feel free to contribute to a more helpful way to describe
>>> the tracing support. I'm sure it would be universally appreciated.
>>
>> The one time I tried to get a patch into anything in Erlang (something
>> that was causing a build under FreeBSD to segfault), it didn't make it
>> into the next release -- even though other people helped get it into
>> shape for submission. Pardon me for being a little leery of the
>> process.
>
> Patching can be pretty hairy, but note that patching the documentation
> is much easier than patching the emulator. ;-)

I'm sure it *can* be hairy, but mine was a single-character change, as
I recall, and it only asked that a C auto array have a dimension of 1
rather than zero (a write to that array unsurprising caused the stack
to be trashed on the platform I was trying to build on, which made
using source-level debugging a non-starter, which made debugging a
total pain in the ass.)

>> And I suggest that you NOT take the opportunity to add any supposed
>> "history" about how AXE implemented Lamport clocks until you can
>> demonstrate that it did. From what I can see in the documentation you
>> pointed to, AXE did not.
>
> Actually, there is some mention of it in the seq_trace docs already:

[snip]

It shows traces "inspired" by AXE, but without the sequence tokens
that would suggest that AXE implemented Lamport clocks -- IF it ever
did.

> More Ericsson history than that is hardly needed, ...

I'd say that history is entirely dispensable. I'd even throw out the
AXE traces, if you could use the space to instead say something about
how useful the trace tokens can be.

> ... but does
> somewhat strengthen my claim that these systems were the main
> source of inspiration.

We're not talking about whether AXE was "the main source of
inspiration". We're talking about whether the seq_trace documentation
(especially given how confusing it is to begin with) should mention
that it implements Lamport clocks, since, after all, (a) it does, and
(b) lots of people who do concurrent programming know what Lamport
clocks are already -- including about half-a-dozen in the Erlang group
(at least if what they cite in their publications is any indication.)

-michael turner