[erlang-questions] Trace-Driven Development

[Michael Turner]

Ulf, you're saying *I'm* off-topic, in how I'm responding to you? Have
you already forgotten how YOU joined this conversation? Here it is
again:

Me:
> I started using seq_trace. It has its own documentation problems, of
> course. For example: since seq_trace is an implementation of Lamport
> clocks, you should *say* somewhere (like, in the first paragraph,
> maybe?) that it's an implementation of Lamport clocks. Don't make it
> sound like your own invention.

You:
"Actually, I believe it is more correct to say that it's based on the
"forlopp trace" thst exists in Ericsson's AXE switches (using the
proprietary language PLEX). Whether forlopp trace was based on Lamport
clocks, I couldn't say. I don't know when it was introduced in the AXE
(Lamport published his paper in 1978, the same year that the first
digital AXE was taken into service), and to what extent it was
informed by Lamport's work."

But now, when I say this:
>> We're talking about whether the seq_trace documentation
>> (especially given how confusing it is to begin with) should mention
>> that it implements Lamport clocks

You respond with this:

"*You* are talking about that, ...."

ALL of the apparent topic drift on this particular point, since your
first response to me, is from you. I've tried to stay on the point: if
you're using Lamport clocks, exposing them in an API, *admitting* (as
Kenneth Lundin did, on this list, in 2007) that you're using them,
then the documentation should say so. To save people time, if nothing
else. But especially so that people who are looking for a Lamport
clock implementation in Erlang will be able to find it easily in
searches.

> I claim that seq_trace implements Lamport clocks "by
> accident", and that it was not the original purpose, nor a complete
> solution to the problem.

(a) You have no evidence of this,

but worse --

(b) all of the evidence I've been able to find is on the other side:
open-literature publications by members of the Erlang group, showing
they are familiar, perhaps even intimate, with Lamport's work.

As to the "original purpose" of "accidentally" implementing Lamport
clocks in seq_trace, what, pray tell, WAS the original purpose of an
"accidental" implementation of them, if it wasn't basically the same
as Lamport's purpose? Just to have some intriguing pairs of numbers to
look at, in otherwise-boring traces?

> or figure out how to build a documentation patch yourself.

If in fact you represent the Ericsson point of view on this issue, I'd
be wasting my time: it'll be rejected for the reason that "we invented
that independently." I'd like to know what the Ericsson point of view
is, before I try something that might be futile.

> Also, if you would be less focused on defending your position, we
> might actually be able to agree on something.

If you would be less focused on defending a position that I'm not sure
the Erlang group actually takes, we might actually be able to agree on
something. So far, only Robert Virding has spoken up on this issue. I
pointed out to him that he's coauthor on a 1993 publication that cites
Lamport's paper -- long before the 1997-8 timeframe you give for
seq_trace requirements acquisition and implementation. He hasn't
responded since.

> ... dealing with FOP - it runs out of heap and throws exceptions
> everytime I try to build the whole docs.

Yes. If I want to make a four-word change, I'll might have to spend
all day getting set up. That was my last experience with trying to get
the Erlang/OTP documentation toolchain working (admittedly on a
slightly unusual platform.)

I'm used to wikis: learn a little markup, and you can fix any problem
almost instantly. Right now, there's someone in Ericsson who could
make my proposed four-word change almost instantly, since they've got
it all set up already.

> [purpose] was to allow for tracing selectively on e.g.
> a call setup sequence in a phone switch*. You shouldn't rely on it for
> any other purpose.

*Boink* The beta-warning in the seq_trace documentation says:

"... the programming interface still might undergo minor adjustments
(possibly incompatible) based on feedback from users."

What little feedback I can find in public suggests that people are
only too happy to have discovered that seq_trace implements Lamport
clocks. And yet you think I shouldn't "rely" on their continued
presence, because dropping them (or changing them to something else
incompatible) is a possible "minor adjustment" to the "interface"? My
idea of a "minor adjustment" would be to define SeqTraceInfo (and
maybe the timestamp) as a record rather than a bare tuple. I would
suggest some such change, except it's really not that important
compared to backward compatibility for other users of seq_trace out
there.

> your refusal to accept that the seq_trace API was not
> meant to implement Lamport clocks, and might well depart from them

If you mean "might well depart" in the *future*, how is that a "minor
interface adjustment"? SerialInfo - the timestamp for Lamport logical
clocks - is all over the API. No, that's a "major algorithm change."

If you mean "might well depart" *now*, why did Kenneth Lundin say it
implements them? If he's wrong, why did nobody in Ericsson correct
him? Lamport clocks are Lamport clocks, regardless of "intent."

> That seq_trace is completely independent of the built-in tracing is also
> misleading.

WTF? Where did I say it was "completely independent"? Where did anyone
say it was? And it seems pretty "built-in" to me. I mean, how do you
do seq_trace (including the Lamport clock counter increments) without
a hack to Send? That's not very far above bare metal.

> ... one of the biggest drawbacks
> of erlang's tracing support is that only one tracer per process at a time
> can be supported.

*Boink*. seq_trace is *part* of "erlang's tracing support". In what
way is it limited to "one tracer *per* process at a time"? It's
limited to one *tracer process* at a time, but, if I'm reading the
documentation right (*sigh*), it can selectively trace an arbitrary
number of tokens through any number of processes they might propagate
into. If you want individual tracer processes on (say) a per-token
basis, or on a per-process basis, I guess you could just use your
single tracer process as a dispatcher for the others. A few more lines
of code, no biggie.

> (And no, the failure to mention Erlang's support for real-time
> tracing in that thesis is more likely to be due to internal rivalry,
> or simply lack of interest in technologies that they can't use
> anyway - Erlang due to past policy issues and AXE since it's
> a legacy system using a weird programming language).

*Double boink*. The thesis was submitted in late 2008. I see Kenneth
Lundin affirming, in mid-2007, that seq_trace implements Lamport
clocks, for an Erlang/OTP that had been open sourced for years by
then. You're saying it's possible that the author of that thesis might
not have been able to find out about seq_trace (or Lamport clocks), or
was not able to use seq_trace (or its Lamport clocks), because of
"internal rivalry" or "past policy issues"???

Things must be a lot weirder in there than I ever suspected. And yet
I'm supposed to be confident that if I submit a patch to the seq_trace
documentation informing users that it implements Lamport clocks, it's
very likely to be taken up?

Please tell me you really meant something else by that last, but just
totally garbled it.

-michael turner

On Tue, Jun 5, 2012 at 4:52 PM, Ulf Wiger <ulf@REDACTED> wrote:
>
> On 5 Jun 2012, at 08:01, Michael Turner wrote:
>
>> We're not talking about whether AXE was "the main source of
>> inspiration". We're talking about whether the seq_trace documentation
>> (especially given how confusing it is to begin with) should mention
>> that it implements Lamport clocks, since, after all, (a) it does, and
>> (b) lots of people who do concurrent programming know what Lamport
>> clocks are already -- including about half-a-dozen in the Erlang group
>> (at least if what they cite in their publications is any indication.)
>
> *You* are talking about that, and in the process, firing off rants
> insulting people who have worked much harder than you to make
> Erlang a solid platform, implying that they are lazy, possibly dishonest
> and ignorant. I claim that seq_trace implements Lamport clocks "by
> accident", and that it was not the original purpose, nor a complete
> solution to the problem. It may be "all it does", but it's not all it should
> do. I don't say that out of ignorance, but because I still recall many of
> the discussions before, during and after the implementation of
> seq_trace.
>
> Meanwhile, you tried to submit a patch once, and since it went
> badly, you can't be bothered to submit a suggestion for how to fix
> this documentation bug that upsets you (but apparently not that
> many others, judging by the lack of sympathetic uproar).
>
> Still, you are making some good points. If you would dispense with
> the sarcasm, chances are people would be more inclined to listen.
> Also, if you would be less focused on defending your position, we
> might actually be able to agree on something.
>
> The reason I reply to you is not because I want to get back at you
> for insulting me (I think even you could agree that I have a track
> record of favoring constructive dialogue here), but because I think
> this is an area that really needs improvement.
>
> There is a clear process in place: either ask the OTP team nicely to
> do this - understanding that they have a ton of higher-priority tasks
> on their plates, or figure out how to build a documentation patch yourself.
> The bar of acceptance is much lower for documentation patches.
> Having patched the docs myself, my own experience is that the biggest
> frustration is dealing with FOP - it runs out of heap and throws exceptions
> everytime I try to build the whole docs. That, and of course the problem
> of writing good documentation in the first place.
>
> I've made this suggestion to you before - I even suggested that your
> documentation patch would be universally appreciated (provided it
> improves on the current docs of course). You have declined. In my book,
> this leaves you with the "ask nicely" option. That, or you can try to pay
> someone money to make your itch their priority. Erlang Solutions actually
> offers that service (and no, I work for neither them nor Ericsson).
>
> But please consider this: seq_trace was never intended as a general
> implementation of Lamport clocks. It is a tracing facility, and the purpose,
> as I've been trying to tell you, was to allow for tracing selectively on e.g.
> a call setup sequence in a phone switch*. You shouldn't rely on it for
> any other purpose. The reference to "beta status" remains there, I would
> guess, because people still find the API and descriptions confusing.
> I think that confusion goes much deeper than just lacking a reference
> to Lamport clocks. Maybe the API should really change into something
> that no longer looks very much like Lamport clocks?
>
> That seq_trace is completely independent of the built-in tracing is also
> misleading. While you can run seq_trace without tracing enabled, the
> match specifications for the trace BIFs have support for seq_trace.
> The seq_trace functionality is also at least partly implemented in
> erl_bif_trace.c
>
> (Actually, not requiring tracing is a great feature, not least if it's to be
> used for other purposes - which it shouldn't in its present state - but
> one can turn it around and note that one of the biggest drawbacks
> of erlang's tracing support is that only one tracer per process at a time
> can be supported. This makes it hard to make broader use of tracing.)
>
> It might well be that the best way forward would be to provide a clean
> API for Lamport clocks - and seq_trace is pretty close to that already -
> and then rework the sequence tracing support (which is badly needed,
> and shamefully under-used) to make clear and intuitive use of it.
> This could achieve two good things: making Lamport clocks available
> in Erlang (not just by accident, but as a documented and supported
> feature), and making sequence tracing more intuitive to use.
>
> But now we're looking at a slightly larger and more difficult task.
>
> I venture a guess that this is very close to what you've been trying
> to say, but your refusal to accept that the seq_trace API was not
> meant to implement Lamport clocks, and might well depart from them
> if it makes it more fit for purpose, reduces the chances that your
> suggestions for improvement will actually be accepted. You need to
> accept the whole picture.
>
> * Just to expand a bit on what the challenge is: in a phone switch,
> or similar, operating under potentially high load and with fairly
> extreme availability demands, running a sequence trace is not
> primarily a question of ordering concurrent events - for practical
> purposes, the trace timestamps are often sufficient for that -
> the big problem is to selectively triggering trace output at just
> the right places and turning it off as soon as it is no longer needed.
> In this respect, Lamport clocks are one brilliant and convenient
> technique that can be built on to solve part of the problem. The
> thesis I linked to lists a few other approaches.
>
> (And no, the failure to mention Erlang's support for real-time
> tracing in that thesis is more likely to be due to internal rivalry,
> or simply lack of interest in technologies that they can't use
> anyway - Erlang due to past policy issues and AXE since it's
> a legacy system using a weird programming language).
>
> BR,
> Ulf W
>
> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
> http://feuerlabs.com
>
>
>