Author: José Valim <jose(dot)valim(at)gmail(dot)com>, Eric Bailey, Radek Szymczyszyn
Status: Draft
Type: Standards Track
Created: 04-Jan-2018
Post-History:

EEP 48: Documentation storage and format

Abstract

This EEP proposes an official API documentation storage to be used by by BEAM languages. By standardizing how API documentation is stored, it will be possible to write tools that work across languages.

Rationale

Currently, different programming languages and libraries running on BEAM devise their own schemas for storing and accessing documentation.
For example, Elixir and LFE provide a h helper in their shell that can print the documentation of any module:

iex> h String
A String in Elixir is a UTF-8 encoded binary.

However, Elixir is only able to show docs for Elixir modules. LFE is only able to show docs for LFE functions and so on. If documentation is standardized, such features can be easily added to other languages in a way that works consistently across all BEAM languages.

Furthermore, each language ends up building their own tools for generating, processing and converting documentation. We hope a unified approach to documentation will improve the compatibility between tools. For instance, an Erlang IDE will be able to show inline documentation for any module and function, regardless if the function is part of OTP, a library or even written in Elixir, LFE or Alpaca.

Note: in this document, the word "documentation" refers exclusively to the API documentation of modules and functions. Guides, tutorials and others materials are also essential to projects but not the focus of this EEP.

Note: This EEP is not about documentation format. It is about a mechanism for storing documentation to make it easier to produce other formats. For example, a tool can read the documentation and produce man pages from it.

Specification

This EEP is divided in three parts. The first defines the two places the documentation can be stored, the second defines the shape of the documentation and the third discusses integration with OTP.

Part 1: the "Docs"storage

There are two main mechanisms in which BEAM languages store documentation: in the filesystem (usually in the /doc directory) and inside .beam files.

This EEP recognizes both options and aim to support both. To look for documentation for a module name example, a tool should:

  1. Look for example.beam in the code path, parse the BEAM file and retrieve the Docs chunk

  2. If the chunk is not available, it should look for "example.beam" in the code path and find the doc/chunks/example.chunk file in the application that defines the example module

  3. If a .chunk file is not available, then documentation is not available

The choice of using a chunk or the filesystem is completely up to the language or library. In both cases, the documentation can be added or removed at any moment by stripping the Docs chunk or by removing the doc/chunks directory.

For example, languages like Elixir and LFE attach the Docs chunk at compilation time, which can be controlled via a compiler flag. On the other hand, projects like OTP itself will likely generate the doc/chunks entries on a separate command, completely unrelated from code compilation.

Part 2: the "Docs" format

In both storages, the documentation is written in the exactly same format: an Erlang term serialized to binary via term_to_binary/1. The term may be optionally compressed when serialized and must follow the type specification below:

{docs_v1,
 Anno :: erl_anno:anno(),
 BeamLanguage :: atom(),
 Format :: mime_type(),
 ModuleDoc :: #{DocLanguage := DocString} | none | hidden,
 Metadata :: map(),
 Docs ::
   [{{Kind, Name, Arity},
     Anno :: erl_anno:anno(),
     Signature :: [binary()],
     Doc :: #{DocLanguage := DocString} | none | hidden,
     Metadata :: map()
    }]} when DocLanguage :: binary(),
             DocString :: binary()

where in the root tuple we have:

For each entry in Docs, we have:

This shared format is the heart of the EEP as it is what effectively allows cross-language collaboration.

The Metadata field exists to allow languages, tools and libraries to add custom information to each entry. This EEP documents the following metadata keys:

Any key may be added to Metadata at any time. Keys that are frequently used by the community can be standardized in future versions.

Part 3: Integration with OTP

The last part focuses on integrating the previous parts with OTP docs, tools and workflows. The items below are suggestions and are not necessary for the adoption of this EEP, neither by OTP nor by any other language or library.

At this point we should consider changes to OTP such as:

FAQ

Q: Why do we have a Format entry in the documentation?

The main trade-off in the proposal is the documentation format. We have two options:

A unified format for documentation gives no flexibility to languages and libraries in choosing how documentation is written. As the ecosystem gets more diverse, it will be unlikely to find a format that suits all. For this reason we introduced a Format field that allows each language and library to pick their documentation format. The downside is that, if the Elixir docs are written in Markdown and a language does not know how to format Markdown, then the language will have to choose to either not show the Elixir docs or show them raw (i.e. in Markdown).

Erlang is in a privileged position. All languages will be able to support whatever format is chosen for Erlang since all languages run on Erlang and will have direct access to Erlang's tooling.

Q: If I have an Erlang/Elixir/LFE/Alpaca library that uses a custom documentation toolkit, will I also be able to leverage this?

As long as the documentation ends up up in the Docs chunk or inside the doc/chunks directory, we absolutely do not care how the documentation was originally written. If you use a custom format, you may need to teach your language of choice how to render it though. See the previous question.

Copyright

This document has been placed in the public domain.