[eeps] [Suspected Spam] Re: EEP 48: Documentation storage and format

[Raimo Niskanen]

On Wed, Jan 10, 2018 at 05:36:56PM +0100, José Valim wrote:
> Thank you Raimo! I have sent a PR based on the discussion in this mailing
> list: https://github.com/erlang/eep/pull/5

Oh!  That is a new way to do it...  Traditionally we have been passing
diffs and new versions on this mailing list.

I have made two inline comments in the PR.

/ Raimo


> 
> 
> 
> 
> 
> *José Valimwww.plataformatec.com.br
> <http://www.plataformatec.com.br/>Founder and Director of R&D*
> 
> On Wed, Jan 10, 2018 at 4:47 PM, Raimo Niskanen <raimo+eeps@REDACTED
> > wrote:
> 
> > I have added your proposal to the repository as EEP 48:
> >
> >     http://erlang.org/eep/eeps/eep-0048.html
> >     https://github.com/erlang/eep/blob/master/eeps/eep-0048.md
> >
> > I changed the quoting of atoms and pathnames to use `code` quoting,
> > changed some strange double and single quote characters into the ASCII
> > ones, and set José as the main author partly because of him sending in the
> > EEP gave me an e-mail address to use obfuscated and partly because he
> > declared the others as co-authors.
> >
> > Thank you for your contribution!
> >
> > / Raimo Niskanen, EEP editor.
> >
> >
> >
> > On Thu, Jan 04, 2018 at 10:59:58PM +0100, José Valim wrote:
> > > This EEP proposes an official API documentation storage to be used by
> > > by BEAM languages.  By standardizing how API documentation is stored,
> > > it will be possible to write tools that work across languages.
> > >
> > > I want to thank Eric and Radek, who co-authored the proposal, as well as
> > > Kenneth, Fred, Tristan and Loïc for their feedback.
> > >
> > > See the attached document.
> > >
> > > *José Valimwww.plataformatec.com.br
> > > <http://www.plataformatec.com.br/>Founder and Director of R&D*
> >
> > >     Author: Eric Bailey, Radek Szymczyszyn, José Valim
> > >     Status: Draft
> > >     Type: Standards Track
> > >     Created: 04-Jan-2018
> > >     Post-History:
> > > ****
> > > EEP XX: Documentation storage and format
> > > ----
> > >
> > >
> > >
> > > Abstract
> > > ========
> > >
> > > This EEP proposes an official API documentation storage to be used by
> > > by BEAM languages.  By standardizing how API documentation is stored,
> > > it will be possible to write tools that work across languages.
> > >
> > >
> > >
> > > Rationale
> > > =========
> > >
> > > Currently, different programming languages and libraries running on
> > > BEAM devise their own schemas for storing and accessing documentation.
> > > For example, Elixir and LFE provide a `h` helper in their shell that
> > > can print the documentation of any module:
> > >
> > >     iex> h String
> > >     A String in Elixir is a UTF-8 encoded binary.
> > >
> > > However, Elixir is only able to show docs for Elixir modules.  LFE is
> > > only able to show docs for LFE functions and so on.  If documentation
> > > is standardized, such features can be easily added to other languages
> > > in a way that works consistently across all BEAM languages.
> > >
> > > Furthermore, each language ends up building their own tools for
> > > generating, processing and converting documentation.  We hope a unified
> > > approach to documentation will improve the compatibility between tools.
> > > For instance, an Erlang IDE will be able to show inline documentation
> > > for any module and function, regardless if the function is part of OTP,
> > > a library or even written in Elixir, LFE or Alpaca.
> > >
> > > **Note**: in this document, the word “documentation” refers exclusively
> > > to the API documentation of modules and functions.  Guides, tutorials
> > > and others materials are also essential to projects but not the focus
> > > of this EEP.
> > >
> > > **Note**: This EEP is not about documentation format.  It is about a
> > > mechanism for storing documentation to make it easier to produce other
> > > formats.  For example, a tool can read the documentation and produce man
> > > pages from it.
> > >
> > >
> > >
> > > Specification
> > > =============
> > >
> > > This EEP is divided in three parts.  The first defines the two
> > > places the documentation can be stored, the second defines the shape of
> > > the documentation and the third discusses integration with OTP.
> > >
> > >
> > > ## Part 1: the "Docs"storage ##
> > >
> > > There are two main mechanisms in which BEAM languages store
> > documentation:
> > > in the filesystem (usually in the /doc directory) and inside ".beam"
> > > files.
> > >
> > > This EEP recognizes both options and aim to support both.  To look for
> > > documentation for a module name "example", a tool should:
> > >
> > >   1. Look for "example.beam" in the code path, parse the BEAM file and
> > >      retrieve the "Docs" chunk
> > >
> > >   2. If the chunk is not available, it should look for "example.beam"
> > >      in the code path and find the "doc/chunks/example.chunk" file in
> > >      the application that defines the "example" module
> > >
> > >   3. If a ".chunk" file is not available, then documentation is not
> > >      available
> > >
> > > The choice of using a chunk or the filesystem is completely up to the
> > > language or library.  In both cases, the documentation can be added or
> > > removed at any moment by stripping the "Docs" chunk or by removing the
> > > "doc/chunks" directory.
> > >
> > > For example, languages like Elixir and LFE attach the "Docs" chunk at
> > > compilation time, which can be controlled via a compiler flag.  On the
> > > other hand, projects like OTP itself will likely generate the
> > "doc/chunks"
> > > entries on a separate command, completely unrelated from code
> > compilation.
> > >
> > >
> > > ## Part 2: the "Docs" format ##
> > >
> > >
> > > In both storages, the documentation is written in the exactly same
> > > format: an Erlang term serialized to binary via term_to_binary/1.  The
> > > term may be optionally compressed when serialized and must follow the
> > > type specification below:
> > >
> > >     {docs_v1,
> > >      Anno :: erl_anno:anno(),
> > >      Language :: atom(),
> > >      Format :: mime_type(),
> > >      ModuleDoc :: binary() | none | hidden,
> > >      Metadata :: map(),
> > >      Docs ::
> > >        [{{Kind, Name, Arity},
> > >          Anno :: erl_anno:anno(),
> > >          Signature :: [binary()],
> > >          Doc :: binary() | none | hidden,
> > >          Metadata :: map()
> > >         }]}
> > >
> > > where in the root tuple we have:
> > >
> > >   * `Anno` - annotation (line, column, file) of the module documentation
> > >     or of the definition itself (see erl_anno)
> > >
> > >   * `Language` - an atom representing the language, for example:
> > >     "erlang", "elixir", "lfe", "alpaca", etc
> > >
> > >   * `Format` - the mime type of the documentation, such as
> > "text/markdown"
> > >     (see the FAQ for a discussion on this field)
> > >
> > >   * `ModuleDoc` - a binary with the documentation or the atom "none"
> > >     in case there is no documentation or the atom "hidden" if
> > >     documentation has been explicitly disabled for this entry
> > >
> > >   * `Metadata` - a map of atom keys with any term as value.  This can be
> > >     used to add annotations like the "authors" of a module, "deprecated",
> > >     or anything else a language or documentation tool may find relevant
> > >
> > >   * `Docs` - a list of documentation for other entities (such as
> > >     functions and types) in the module
> > >
> > > For each entry in `Docs`, we have:
> > >
> > >   * `{Kind, Name, Arity}` - the kind, name and arity identifying the
> > >     function, callback, type, etc.  The official entities are:
> > `function`,
> > >     `type` and `callback`.  Other languages will add their own. For
> > >     instance, Elixir and LFE may add `macro`
> > >
> > >   * `Anno` - annotation (line, column, file) of the module documentation
> > >     or of the definition itself (see erl_anno)
> > >
> > >   * `Signature` - the signature of the entity.  It is is a list of
> > >     binaries. Each entry represents a binary in the signature that can
> > >     be joined with a whitespace or a newline.  For example,
> > >     `["binary_to_atom(Binary, Encoding)", "when is_binary(Binary)"]`
> > >     may be rendered as as a single line or two lines. It exists
> > >     exclusively for exhibition purposes
> > >
> > >   * `Doc` - a binary with the documentation or the atom "none"
> > >     in case there is no documentation or the atom "hidden" if
> > >     documentation has been explicitly disabled for this entry
> > >
> > >   * `Metadata` - a map of atom keys with any term as value
> > >
> > > This shared format is the heart of the EEP as it is what effectively
> > > allows cross-language collaboration.
> > >
> > > The `Metadata` field exists to allow languages, tools and libraries to
> > > add custom information to each entry.  This EEP documents the
> > > following metadata keys:
> > >
> > >   * `authors := [binary()]` - a list of authors as binaries
> > >
> > >   * `cross_references := [module() | {module(), {Kind, Name, Arity}}]` -
> > >     a list of modules or module entries that can be used as cross
> > >     references when generating documentation
> > >
> > >   * `deprecated := binary()` - when present, it means the current entry
> > >     is deprecated with a binary that represents the reason for
> > >     deprecation and a recommendation to replace the deprecated code
> > >
> > >   * `since := binary()` - a binary representing the version such entry
> > >     was added, such as "1.3.0" or "20.0"
> > >
> > > Any key may be added to Metadata at any time.  Keys that are frequently
> > > used by the community can be standardized in future versions.
> > >
> > >
> > > ## Part 3: Integration with OTP ##
> > >
> > > The last part focuses on integrating the previous parts with OTP docs,
> > > tools and workflows.  The items below are suggestions and are not
> > > necessary for the adoption of this EEP, neither by OTP nor by any other
> > > language or library.
> > >
> > > At this point we should consider changes to OTP such as:
> > >
> > >   * Distributing the `doc/chunks/*.chunk` files as part of OTP and
> > >     changing the tools that ship with OTP to rely on them. For example,
> > >     "erl -man lists" could be changed to locate the "lists.chunk" file,
> > >     parsing the documentation out and then converting it to a man page
> > >     on the fly.  This task may require multiple changes, as OTP stores
> > >     documentation on XML files as well as directly in the source code.
> > >     `edoc` itself should likely be augmented with functions that spit
> > >     out `.chunk` files from the source code
> > >
> > >   * Adding `h(Module)`, `h(Module, Function, Arity)`, and similar to
> > >     Erlang’s shell to print the documentation of a module or of a
> > >     given function and arity. This should be able to print docs any
> > >     other library or language that implements this proposal
> > >
> > >
> > >
> > > FAQ
> > > ===
> > >
> > > *Q: Why do we have a Format entry in the documentation?*
> > >
> > > The main trade-off in the proposal is the documentation format.  We have
> > > two options:
> > >
> > >   * Allow each language/library/tool to choose their own documentation
> > >     format
> > >   * Impose a unified documentation format on all languages
> > >
> > > A unified format for documentation gives no flexibility to languages and
> > > libraries in choosing how documentation is written.  As the ecosystem
> > > gets more diverse, it will be unlikely to find a format that suits all.
> > > For this reason we introduced a Format field that allows each language
> > > and library to pick their documentation format.  The downside is that,
> > > if the Elixir docs are written in Markdown and a language does not know
> > > how to format Markdown, then the language will have to choose to either
> > > not show the Elixir docs or show them raw (i.e. in Markdown).
> > >
> > > Erlang is in a privileged position.  All languages will be able to
> > > support whatever format is chosen for Erlang since all languages run on
> > > Erlang and will have direct access to Erlang's tooling.
> > >
> > > *Q: If I have an Erlang/Elixir/LFE/Alpaca library that uses a custom
> > > documentation toolkit, will I also be able to leverage this?*
> > >
> > > As long as the documentation ends up up in the "Docs" chunk or inside
> > > the `doc/chunks` directory, we absolutely do not care how the
> > > documentation was originally written.  If you use a custom format,
> > > you may need to teach your language of choice how to render it though.
> > > See the previous question.
> > >
> > >
> > >
> > > Copyright
> > > =========
> > >
> > > This document has been placed in the public domain.
> > >
> > >
> > >
> > > [EmacsVar]: <> "Local Variables:"
> > > [EmacsVar]: <> "mode: indented-text"
> > > [EmacsVar]: <> "indent-tabs-mode: nil"
> > > [EmacsVar]: <> "sentence-end-double-space: t"
> > > [EmacsVar]: <> "fill-column: 70"
> > > [EmacsVar]: <> "coding: utf-8"
> > > [EmacsVar]: <> "End:"
> > > [VimVar]: <> " vim: set fileencoding=utf-8 expandtab shiftwidth=4
> > softtabstop=4: "
> > > _______________________________________________
> > > eeps mailing list
> > > eeps@REDACTED
> > > http://erlang.org/mailman/listinfo/eeps
> >
> >
> > --
> >
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > _______________________________________________
> > eeps mailing list
> > eeps@REDACTED
> > http://erlang.org/mailman/listinfo/eeps
> >

> _______________________________________________
> eeps mailing list
> eeps@REDACTED
> http://erlang.org/mailman/listinfo/eeps


-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB