[erlang-questions] Erlang documentation -- a modest proposal

[Kenneth Lundin]

I have followed this thread with interest there has been many good
suggestions and comments and I will in a separate mail comment the OTP
teams thinking about this. But first I must put some comment on Joes mail.

On Fri, Sep 23, 2016 at 10:13 AM, Joe Armstrong <erlang@REDACTED> wrote:

> I have a few comments about the documentation:
>
> This answers a few question that keep arising, and has a few of my own
> suggestions.
>
> Q: Who was the documentation written for?
>
> A: Ericsson Internal usage
>
>    The original documentation made the following assumptions
>
>    - The readers would be internal Ericsson programmers who knew Erlang
>    - If the programmers had a problem understanding things they would ask
>      us directly
>
>    So we favored a terse declarative style
>
Since I joined the project (in 1996) I think the documentation is meant to
be a reference manual which means it is
not written for total beginners in programming. You are actually supposed
to know something about Erlang and or functional programming when you read.

>
> Q: How is the documentation structured
> A: It's all in XML
>
>    The idea was that all documentation would be generated from the XML
>    There are current about 1000 XML files containing about 10MB of text
>
>    The XML sources are in the distribution source tree.
>    So for example the documentation for lists.erl is in the file
>    <Dir>/lib/stdlib/doc/src/lists.xml (<Dir> is where the system was
> unpacked)
>

The idea is that the documentation is part of the SW delivery, it is in the
same source code tree. Now it is in the same repository organized per OTP
application.

The principle is also that there is only one source of documentation which
can be presented in many ways (generated by tools). The documentation
should be written without thinking about layout and presentation. That part
should be handled by the tools generating the different views.

>
> Q: How is the documentation produced
> A: Programmatically
>

There is an otp application erl_docgen which is used for this. It takes the
XML input and translates through
several steps into unix man pages, html and pdf.

>
>    Various programs munge the XML inputs (and the sources) trying to make
>    web-sites and PDF
>
>    We could use some help here - really - the PDF is produced with
> some XSLT magic
>    and the web site with goodness knows what.
>

The xml is translated to html by means of some preporcessing in Erlang and
then an xslt transform executed with xsltproc which is pretty standard on
any Linux system. The html is bundled together with a .css and a javascript
for the left side navigation. Everything is passive, no active server is
needed.
It is "easy" to generate something nicer here without any impact on the doc
sources. More about this in a separate mail.

>
> So what's wrong and what could we do?
>
> 1) Things are not beautiful
>
>    I agree - the documentation is not typographically beautiful
>
>    Personally I like PDF - I took a look and found this:
>
>    http://www.latextemplates.com/template/the-legrand-orange-book
>
>    It would be nice to transform the documentation into this form
>    for printing and reading
>

I am actually thinking that we don't need PDF at all. HTML should be enough.
Man pages can also be questioned since they are limiting when it comes to
format.
Would be interesting to know how much use there is of our man pages?.

>
> 2) The HTML documentation could be improved and made more interactive
>
>    I have at various times written 'erl_to_html' that makes HTML pages out
> of
>    erlang - it would be nice to browse your own code, clicking on function
> names
>    to follow the code or to unfold the documentation.
>
The html can be much improved. We are not experts on html and javascript,
so help her is welcome. I have looked
at the Elixir documentation generated with their ex_doc. The same or
similar layout would be easy and suitable to use for Erlang as well and the
communities are overlapping anyway so it would perhaps be a good idea to
have a similar layout.
I think it looks quite pretty.

>
>    On-line documentation can have things like type spec in hidden folds
>    which open in place when you click on them. Or possibly a tiddly-wiki
> like
>    interface.
>
>    http://tiddlywiki.com/ might inspire a new interface
>
> 3) There is no official proof-reading quality control mechanism
>
>    I tried on several occasions to hire a technical author - management
> always
>    views this as a waste of money - so this never happened.
>

We have technical writers engaged since some years now. But they can only
look after the language and the
consistency of naming things etc. They can not add new text or examples.

>
>    Most of the documentation is written by people who are not
>    native English speakers. They could use some help.
>
>    By contract my Erlang book had
>
>      - Me (who wrote it)
>      - Dave Thomas (my editor) (he was great)
>      - A Proof Reader (By proof reader I mead one of these super-human
> beings
>        who can spot a spelling error at 10,000 meters from your text,
>        while reading upside down by candle light
>      - An Indexer (did you know there were professional Indexers?)
>      - A typographer (who worried about
> https://en.wikipedia.org/wiki/Widows_and_orphans)
>
>    Have you every heard *anybody* complaining about widows and orphans
>    in the Erlang printed documentation? I had to rephrase several
>    paragraphs to avoid nasty widows and orphans.
>
> 4) There are no examples
>
>    I agree there should be zillions of examples.
>
>    Rather than adding them to the existing XML tree I'd propose a new
>    DTD for examples which could be cross-linked into the
>    documentation.
>
I agree that there should be more examples and I think they should be
placed in the existing xml-files or
in a separate file per module, but still treated as documentation source.
Then there can be several alternatives for how they are presented, but this
is a tooling or html output question.


>
>
> 5) There are no construction examples
>
>    Examples show "the finished work" but not "how you got there"
>
>    In my book I often shown really short functions in 10 lines or so
>    of code.  But I don't discuss how I got to those ten lines.
>
>    I show the 10 lines and explain what each lines does.
>
>    I do not explain how I wrote the first line - I might have googled
>    a bit done some research. In a 50 line program I'll probably have
>    tested it as I grew it - testing 10 lines at a time.
>
>    How we grow programs from small seeds is not explained.
>
This does not belong to the reference manual. But somewhere else, I don't
know where.

>
> 6) The barrier for entry for fixing a typo is HUGE
>
>    If I see a typo (a simple spelling error) in the on-line documentation
>    the barrier of entry for fixing it is HUGE (sorry for SHOUTING).
>
>    Download the entire distribution - unpack (240 MBytes) find the
>    typo (where the heck is it?) - fix it - make a push request ....
>
>    Do I really have to download hundreds of Megabytes to fix a one
>    character typo?
>
I agree that it should be simple to correct a simple typo, and it seems to
be a way through github , we will explore that
and describe it. I link to the doc source at github could for example be
generated into the html representation of a module. Then it would be
obvious in which file to fix the error.

>
>    I keep pointing people to the web site for "real world Haskell"
>    http://book.realworldhaskell.org/read/functional-programming.html
>
>    The barrier for entry to fixing a mistake in the text is minimal.
>    I actually like the RWH model - the reader adds a comment, the author
> fixes
>    the bug.
>
>    When writing books I would not like other people fixing my typos, I
>    want many eyes to help be find the typos - then I'll fix them? Why
>    is this? Because I'm ultimately responsible for what I write -
>    Sometimes (not often) I might want to totally rewrite a section
>    based on a single typo.
>
>    If somebody could figure out how to automate fixing typos it would be
> great.
>    [the problem is to find the XML that needs to be edited, given that the
> error
>     was found in a generated document - I guess if each paragraph has
> a hidden GUID
>     <<WHICH I'VE BEEN RANTING ON ABOUT FOR YEARS>> it would be trivial]
>
>    <aside>All paragraphs should have GUIDs and be stored in a global
> content
>    adressable store (why GUIDs? - we could use the SHA as a key but
> they we could not
>    edit the paragraph :-) </aside>
>
>
> 7) The erlang mailing list has loads of examples
>
>    The erlang mailing list has thousands of useful examples
>    but they are not extracted, edited sorted and classified.
>
>    There are issues of copyright and attribution here - but I think
>    it would be useful to extract some of the better postings to this list
>    and move them into the documentation tree.
>
>
> Making beautiful readable useful documentation is very difficult and
> needs love care and attention to detail - it's an unappreciated form
> of literature.
>

/Kenneth, Erlang/OTP Ericsson

>
> On Fri, Sep 23, 2016 at 8:40 AM, Loïc Hoguin <essen@REDACTED> wrote:
> > On 09/23/2016 05:18 AM, Kenneth Lakin wrote:
> >>
> >> Speaking of the User's Guide... -ideally- someone with no Erlang
> >> experience should first be reading the Reference Manual User's Guide
> >> along with the Getting Started Guide (or LYSE or something) to pick up
> >> the fundamentals, rather than the various module Reference Manuals. In
> >> my experience, reference manuals exist to quickly provide useful
> >> information to people who are already familiar with a language. They
> >> shouldn't be language primers, as primers serve another purpose and
> >> should be in another document.
> >
> >
> > That's the thing: you have no control over what document people will read
> > first. You also have no control over what people will *remember*.
> >
> > When first coming to the language, most people will
> > forget/misremember/misunderstand most of what they read until they get
> some
> > practice; and that's when a good function reference with all details and
> > examples is helpful.
> >
> >
> > --
> > Loïc Hoguin
> > http://ninenines.eu
> > Author of The Erlanger Playbook,
> > A book about software development using Erlang
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160923/7bdd2c1a/attachment.htm>