Fix for A=<<1>>

[Raimo Niskanen]

You are of course right, but:

Sometimes I and many others do not use spaces to separate the symbols. 
The best example I can think of is:
	Rec#rec{length=0,state=init,count=0,buf=<<>>}
where I do _not_ want spaces.

Size comparision of binaries I think is so uncommon that the programmer 
that thinks "A =< <<1>>" does not write "A=<<<1>>". It is to ambiguous 
to the eye. Especially size comparisions of binaries must be _very_ 
uncommon. The programmer that skips as many spaces between tokens as 
possible is begging for trouble.

Equality comparisions of binaries I think is also rather uncommon, but 
note that "A==<<1>>" is scanned correctly(tm) and "A==<1>>" gives an 
error since scanning of '==' is eager. Remains typing "A=<<<1>>" when 
meaning "A==<<1>>", but that is a double fault.

About the code comments: more precisely, which (kind of) comments need 
to be fixed?

About including the state in the fun: I guess more/6 could be change 
into something like:

more(Cs, Pos, State, Fun) ->
     {more,{Cs,Pos,State,Fun}}.

And tokens/3 into:

tokens([], Chars, Pos) ->
     tokens({[],Pos,io,
            fun (Cs,P,State)->
                scan(Cs, [], [] Pos, State})
            end, Chars, Pos);
tokens({Cs,Pos,eof,_Fun}, eof, _) ->
     {done,{eof,Pos},Cs};
tokens({Cs,Pos,_State,Fun}, eof, _) ->
     Fun(Cs++eof, eof);
tokens({Cs,Pos,State,Fun}, Chars, _) ->
     Fun(Cs++Chars, State).

note that Cs, Pos, State and Fun are still needed in tokens/3. The fun 
can only contain Stack, Tokens and Pos.


Then the calls to more/6 would have to change into:
scan(">"=Cs, Stack, Toks, Pos, State) ->
     more(Cs, Pos, State, fun (C, S) ->
                              scan(C, Stack, Toks, Pos, S)
                          end);

And these calls are rather many so it will clutter the code compared to:

scan(">"=Cs, Stack, Toks, Pos, State) ->
     more(Cs, Stack, Toks, Pos, State, fun scan/5);

By passing two (as far as I can see they are only 2: Stack and Tokens) 
unneseccary arguments to more/6 I can use the "fun scan/5" notation 
which improves readability.

Or did you have a smarter change in mind?

/ Raimo



Robert Virding wrote:
> The problems are at different levels:
> 
> 1. There is a fundamental difference in scanning atoms/strings and =<<<<<.
> You have basically introduced infinite look-ahead (at least to the end of
> the file) to try and determine what the first token actually is, for
> everything else there is one character look-ahead. (As is the grammar which
> is one LALR1)
> 
> 2. There are at least three different "one key-press typos" which can give
> "A=<<1>>":
> 
> A==<<1>>
> A=<<<1>>
> A= <<1>>
> 
> Two tests and a match. How can you so so certain what I meant as to choose
> one? Introducing a match when I really meant a test will introduce a
> fundamental semantic change to the code, both as to return value and to the
> error case. Also if I had mistyped the variable name then this *really*
> changes the meaning of the code. And it does it automagically, completely
> silently and in a way which can make it extremely difficult for the
> programmer to find!
> 
> 3. Nothing should try to correct its input, especially not something as
> fundamental as the scanner. Especially when the "correction" is not
> unambiguous. Just because you now don't think the chances of meaning
> something else is slim doesn't mean other people think the same way. Or that
> you might not think so in the future. :-)
> 
> 4. I personally always (or almost always) use spaces to separate my symbols
> so I don't really see what the problem is.
> 
> My basic premise is that you can not add an "improver" which works silently
> and is only sometimes correct. Will you take the responsibility when this
> generates an serious, invisible error in someones code?
> 
> The trouble with writing code at this level is that you always have to try
> and handle the case when users do things which you had assumed was so stupid
> or strange,even though it is legal,  that no one would do it, in this case
> mean "A=<<<1>>" when they  wrote "A=<<1>>". I remember that a relatively
> early version of the JAM compiler could not handle the case when you did a
> send as an argument to a function to get the message in as an argument,
> (foo(X ! <big evaluation>, ...)). The premise was that send is used for side
> effects not return values. Of course someone did just this and it crashed.
> 
> Robert
> 
> P.S. Liked the new code. You need to fix the comments. Can't the fun be
> generated each suspend and include the state? I think it would make things
> easier.
> 
>