[erlang-questions] leex and case-insensitive match

[Robert Virding]

The lexer file would not become smaller if you had some syntax for saying
you wanted a case independent match. The way the resulting scanner works
means that you would basically get the same set of lexer rules. This a
result of the lexer being completely deterministic. It's all in the dragon
book. :-)

Robert


On 4 June 2015 at 15:33, Knut Nesheim <knutin@REDACTED> wrote:

> At the moment I'm using the first solution you propose (and proposed
> by Joel in the old post). It works out fine. It's cumbersome to write
> and the lexer file becomes huge in my case. The generated module is 8k
> lines of code. Otherwise, I don't have any problems.
>
> What I would like to do in a perfect world is to use the "i" flag
> found in some regex implementations:
>
>     /select/i     : {token, {select, TokenLine, TokenChars}}
>
> Regards
> Knut
>
> On Thu, Jun 4, 2015 at 8:15 PM, Jesper Louis Andersen
> <jesper.louis.andersen@REDACTED> wrote:
> >
> > On Thu, Jun 4, 2015 at 7:06 PM, Knut Nesheim <knutin@REDACTED> wrote:
> >>
> >> For my use case "select", "SELECT", "SeLeCt" means exactly the
> >> same.
> >
> >
> > I'm not sure I understand this:
> >
> > [jlouis@REDACTED ~]$ cat z.xrl
> > %% Test
> >
> > Definitions.
> >
> > Rules.
> >
> > [sS][eE][lL][eE][cC][tT] : {token, select}.
> >
> > Erlang code.
> >
> > %% Nothing
> >
> > in an erl shell:
> >
> > 4> leex:file("z").
> > {ok,"./z.erl"}
> > 6> c("z.erl").
> > {ok,z}
> > 9> [z:string(Str) || Str <- ["select", "SELECT", "SeLeCt"]].
> > [{ok,[select],1},{ok,[select],1},{ok,[select],1}]
> >
> > Note how the token parses as the same thing.
> >
> > [sS][eE][lL][eE][cC][tT] : {token, {identifier,
> > string:to_lower(TokenChars)}}.
> >
> > should also work, if select is not a keyword, but you want to handle it
> as
> > an identifier.
> >
> > some Regex engines provides convenience notation for [sS], [eE] etc, but
> it
> > tends to be rare enough for keywords that you skip it. Another path
> through
> > the game is the following age-old lexer hack:
> >
> > [jlouis@REDACTED ~]$ cat y.xrl
> > %% Test
> >
> > Definitions.
> >
> > ALPHA = [a-zA-Z]
> >
> > Rules.
> >
> > {ALPHA}* : {token, analyze_alpha(string:to_lower(TokenChars))}.
> >
> > Erlang code.
> >
> > %% Match keywords, and if no keyword matches, regard the token as an
> > identifier
> > analyze_alpha("select") -> select;
> > analyze_alpha(Otherwise) -> {identifier, Otherwise}.
> >
> > 13> leex:file("y").
> > {ok,"./y.erl"}
> > 14> c("y.erl").
> > {ok,y}
> > 15> [y:string(Str) || Str <- ["select", "SELECT", "SeLeCt"]].
> > [{ok,[select],1},{ok,[select],1},{ok,[select],1}]
> >
> > Do any of these work for you?
> >
> > --
> > J.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150604/874a8277/attachment.htm>