[erlang-questions] extending the scanner and parser

[Tamas Nagy]

Hi Vlad!

Check out RefactorErl and contact the developers if you need something  
which is not available. I'm pretty sure they can help.

It provides all the functionality which is mentioned in your e-mail.  
(These are all needed for refactoring as well to be able to retain the  
original layout). Furthermore it has an API to retrieve this  
information. So you do not have to hack it to get the data needed. :)

The website:
http://plc.inf.elte.hu/erlang/

Regards,
	Tamas

On 3 Oct 2008, at 12:13, Vlad Dumitrescu wrote:

> Hi,
>
> Tools that process Erlang source code need more detailed information
> about the lexical tokens than those provided by the scanner in the
> standard distribution.  For example, ErlIDE needs the character offset
> in the file and also the textual representation of the token.  It also
> adds as an optimization the length of the textual token.  Wrangler
> uses column information.
>
> These tools also need to be able to reconstruct the source code from
> the token list.  For this to be possible, it must be possible to ask
> the scanner to return even whitespace and comments.  It must also be
> possible to know the exact textual representation of the token, for
> example an integer valued 42 may have been written as "42" or as
> "16#2A" in the source file.
>
> In a similar way, the parse tree may need to know for each construct
> where it starts and where it ends in the source, so it needs to keep
> track of the underlying tokens.
>
> The way these tools get the extra information is by making a copy of
> erl_scan and erl_parse and modifying them to suit their purposes.
> This is a big maintenance headache and it limits interoperability
> between these tools, but currently the only way to solve the problem.
>
> Therefore I would like to suggest to extend the standard scanner and
> parser to support such extra information. If there will be a consensus
> that this is a Good Thing, I will happily provide a reference
> implementation in cooperation with any interested parties.
>
> If there is not enough general interest, I would still like to call
> out to all those who develope source handling tools and see if we
> could create a common library to share.
>
> best regards,
> Vlad
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions