[erlang-questions] Erlang documentation -- a modest proposal

Vlad Dumitrescu <>
Mon Sep 26 22:49:03 CEST 2016


Hi Joe,

On Mon, Sep 26, 2016 at 10:03 PM, Joe Armstrong <> wrote:

> 4) The Erlang parser should be changed to  exactly
> reproduce the source.
> Right now the parse tree of correct erlang has all the comments
> and white space removed. I'd suggest attaching the comments to the
> next following token (for example {atom,Line,theAtom} should become
>     {atom, Line, theAtom, "the preceding comments and white space"}
> It should be possible to *exactly* reconstruct the input from the parse
> tree.
>
> <aside> - in the first erlang all the different ways of writing an integer
> ended up as the same token. So writing 16#fc was the same as writing the
> integer 252 and tokenized as {integer,Line,252} - the tokenizer threw
> away the exact input so it was impossible to reconstruct the source
> from the token stream. Now it's better the 16#fc is tokenized as
> {integer,[{location,{Line,Col}},{text,"16$fc}], 252} - but comments
> and white space are not
> retained in the parse tree.
>
> Note: that change the parse tree is *not* a simple hack - all tools that
> depend upon the parse tree have to be changed.
> </aside>
>

This is what I set out to implement with 'sourcer' (
https://github.com/erlang/sourcer). I am actually aiming even higher: keep
track of macros and -ifdefs and the possibly different structure of a
module when considering them. Also, I want to be able to parse code that is
in the process of being edited (only possible if knowing the latest
parseable state of the file). On the bright side, this parser need not
replace erl_parse. The compiler and most tools don't need all this extra
detail. It would be good to be able to not implement _everything_ from
scratch, but I couldn't find a way to do all this than with a hand-written
parser. Ideas, feedback, suggestions, improvements and rotten tomatoes are
welcome.

regards,
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160926/c431c527/attachment.html>


More information about the erlang-questions mailing list