Lexing and parsing, was Re: plain_fsm - for beginners and purists

[Ulf Wiger]

On Wed, 11 Feb 2004 10:36:46 -0800 (PST), Thomas Lindgren 
<thomasl_erlang@REDACTED> wrote:

> Interesting idea, but wouldn't the Module have to be
> an entire tokenizer? Just processing a stream of
> tokens (e.g., [... bang, bang, ...] => [..., bangbang,
> ...]) might lead to confusion: how to separate "!!"
> from "! !"?

One easy way to address this would be to have the tokenizer
also account for columns - not just line numbers.

tok_xform([{'!', {L,Col}}, {'!',{L,Col1}}|T]) when Col1==Col+1 ->
    ... % bangbang;

It would be _almost_ backward compatible.  (:

/Uffe

-- 
Ulf Wiger, Senior System Architect
EAB/UPD/S

This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.

E-mail including attachments is susceptible to data corruption, interruption, unauthorized amendment, tampering and viruses, and we only send and receive e-mails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.