[erlang-questions] Erlang documentation -- a modest proposal

[Joe Armstrong]

On Mon, Sep 26, 2016 at 10:49 PM, Vlad Dumitrescu <vladdu55@REDACTED> wrote:
> Hi Joe,
>
> On Mon, Sep 26, 2016 at 10:03 PM, Joe Armstrong <erlang@REDACTED> wrote:
>>
>> 4) The Erlang parser should be changed to  exactly
>> reproduce the source.
>> Right now the parse tree of correct erlang has all the comments
>> and white space removed. I'd suggest attaching the comments to the
>> next following token (for example {atom,Line,theAtom} should become
>>     {atom, Line, theAtom, "the preceding comments and white space"}
>> It should be possible to *exactly* reconstruct the input from the parse
>> tree.
>>
>> <aside> - in the first erlang all the different ways of writing an integer
>> ended up as the same token. So writing 16#fc was the same as writing the
>> integer 252 and tokenized as {integer,Line,252} - the tokenizer threw
>> away the exact input so it was impossible to reconstruct the source
>> from the token stream. Now it's better the 16#fc is tokenized as
>> {integer,[{location,{Line,Col}},{text,"16$fc}], 252} - but comments
>> and white space are not
>> retained in the parse tree.
>>
>> Note: that change the parse tree is *not* a simple hack - all tools that
>> depend upon the parse tree have to be changed.
>> </aside>
>
>
> This is what I set out to implement with 'sourcer'
> (https://github.com/erlang/sourcer). I am actually aiming even higher: keep
> track of macros and -ifdefs and the possibly different structure of a module
> when considering them. Also, I want to be able to parse code that is in the
> process of being edited (only possible if knowing the latest parseable state
> of the file). On the bright side, this parser need not replace erl_parse.
> The compiler and most tools don't need all this extra detail. It would be
> good to be able to not implement _everything_ from scratch, but I couldn't
> find a way to do all this than with a hand-written parser. Ideas, feedback,
> suggestions, improvements and rotten tomatoes are welcome.

Great - I shall take a look.

/Joe

>
> regards,
> Vlad
>