[erlang-questions] Is there a good source for documentation on BEAM?

[Joe Armstrong]

On Tue, May 29, 2012 at 12:08 AM, Richard O'Keefe <ok@REDACTED> wrote:
>
> On 29/05/2012, at 3:25 AM, Thomas Lindgren wrote:
>> BEAM docs are not needed to produce a second source implementation, as shown by several examples. (1)
>
> People were not asking for BEAM documentation in order to produce
> a second source implementation.  That's a red herring.  People
> ask for BEAM documentation in order to understand and perhaps
> revise the main implementation.
>
> One thing that Open Source is about is about community contributions
> and it is awfully hard to contribute to something that is
> under-documented.
>>
>> Also, there has so far been little practical interest shown by Erlang users in such a second source. So implementation efforts may be in vain. (3)
>
> Again, the red herring.  This is *NOT* about second implementations.
> All of the additional implementations I've ever heard of used their
> own back end, *not* BEAM, and would *not* have received any great
> benefit from BEAM being documented.
>
> There is detailed documentation available for Lua and Icon and the
> WAM and several other VMs around, so there's not even any great
> advantage in learning about VMs from BEAM.
>
>> My personal view, at least, is that most of the difficulty in "keeping up with The Real Thing", Erlang/OTP, would be not in reproducing BEAM but in writing a fully compatible implementation tracking the rest of the runtime, ERTS. (2)
>
> Well, not _just_ that, but that's a big part of it.
>
>> So, is there a _practical_ case for doing these docs? In particular, will the effort result in useful external contributions that outweigh time spent? Not at all clear to me.
>
> Bearing in mind that "BEAM documentation" and "second system implementation"
> are about as irrelevant to each other as any two topics about the same
> language can be, what _might_ we realistically expect from BEAM documentation?
>
> (1) There are now several languages implemented on top of Erlang.
>    Some are interpreted in Erlang, some are compiled to Erlang ASTs,
>    and some are compiled to BEAM.  Compiling to the BEAM is *much*
>    harder than it needs to be because of the current undocumentation.
>
>    This isn't *replacing* Erlang/OTP but *augmenting* it.
>
> (2) I've proposed a number of minor but handy extensions to Erlang
>    syntax that lack a reference implementation because I have no idea
>    what the compiler should generate for them; these could use the
>    existing BEAM.
>
> (3) There are existing things that could be compiled more efficiently.
>    I'm thinking here in particular of list comprehension.  I'm not
>    sure if the improved translation can be done with the existing
>    BEAM or if minor extensions would be needed, because the
>    undocumentation for the BEAM does not make clear any range
>    limits or other restrictions on BEAM instructions.
>
> (4) There are even bigger changes, like frames, where even estimating
>    the scope of the changes is hard because of the undocumentation.
>
> But above all you are making an assumption which I utterly reject,
> namely that documentation is a COST and ONLY a cost, that producing
> documentation provides no DIRECT benefits to the people writing the
> documentation.
>
> On the contrary,
>  - you may find defects as you document
>  - you may be able to structure the documentation so that
>   test cases can automatically be extracted
>  - you may in the very act of explaining a limitation see
>   how it can be overcome
>  - you may be able to extract parts of the implementation
>   automatically from the documentation
> and I could go on.
>
> There's a slogan I learned from a business textbook:
>  find the indispensable man and fire him!
> There was something I found unsettling: we were told in this
> thread that there wasn't any need for documentation because
> the Erlang/OTP maintainers had enough people who _knew_ this
> stuff.
>
> One of my colleagues here was managing a software project
> once; a key employee went on holiday and was murdered in a
> far country.  A former student of mine was starting up a
> new company with me as a consultant, and being a worse
> driver than he thought, drove at speed into a tree.  The
> tree survived.  One of my former colleagues, a very
> intelligent and likable guy, was cycling down a Melbourne
> street and got knocked over by a hit-and-run driver.  The
> result was head injury and someone who could dress himself
> but couldn't program if his life depended on it.  When I
> was at Quintus, the founder who wrote the compiler and was
> the only person who really understood it quit and was never
> heard from again.
>
> Fergus O'Brien supervised an MSc on "Organisational
> Forgetting" and the fact of organisational forgetting has
> haunted the fringes of my mind ever since.
>
> People die.  People get head injuries.  People quit.
> Documents get lost.  (If you ever find an architecture-
> of-the-ICL-2900 manual, I'd like to see it.)  Even if
> things _are_ written down, organisations forget _where_.
> If they aren't written down, they WILL be forgotten
> sooner or later.
>

Brilliant! and well argued. Thank you Richard. This last argument is
excellent brain-food.


<aside> I like the way many Erlang discussion threads turn into
meta-discussions about the underlying
problems .. great stuff </aside>

So "stuff" should be well documented precisely because one day the creator
of the stuff will get bored or die and the encompassing organisation
will collectively
forget how it works.

I've notice a lot the "some other guy knows this stuff" phenomena -
I've been chasing a particular
problem for months. I always get the "I don't know but X knows" - I
ask X and they refer to X1 etc.
But X1 does not know ... I'm running out of leads. It appears that
*nobody* knows.

Fortunately the BEAM is not yet there. I know who knows - and when I
ask them they *do* know
so I just hope the entire OTP group aren't on the same plane to an
Erlang conference in a far away
land ...

I conclude it should be documented.

The next question is "priority" ... in the real world we juggle
priorities. The case *is* made that
we should document the BEAM but is this more or less important than
implementing frames etc?
As always, a tricky question...


/Joe


>
> People
> Well, my colleague Andrew Trotman once had a key employee

 and ... ??


>
>
>
>
> We
>>
>> Hope it helps.
>>
>>
>> Best regards,
>> Thomas
>>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions