Fix for A=<<1>>

[Robert Virding]

Doesn't all this work with a special case just to handle a general case tend to indicate that the original special case was probably wrong.

VERY IMPORTANT
Actually this special case means that tokens are no longer scaned in the same way. Everywhere else tokenising is eager, you collect as many characters as you can to mke a token EXCEPT HERE. Why not add other fantastic special cases:

"caseVar" should be scaned as "case Var"
"ifVar" should be scanned as "if Var"
How about any word starting with receive/case/if should be split automatically as this is obviously what the programmer intended. As should any word ending in end. Obvious. Actually any token with if/case/receive/end should be split around them.

Need I go on? There is no fundamental difference between these examples and the original one!!

So this change should not be added for two main reasons:

1. You are introducing an inconsistency and inconsistencies are always bad.
2. You are trying to guess what the user actually meant and at the tokenising stage you have no idea of context. When I wrote "A=<<1>>" I might have actually meant to write "A =< <<1>>" so changing it automagically to "A=<<1>>" introduces a fundamental semantic change to my code.

Can you GUARANTEE that this change is always what the programmer intended? ALWAYS? If not then you can't make this change.

Sorry if sound a bit harsh but someone has to be the devil's advocate.

Robert

P.S. Raimo how did you get the speed up. I know the original was coded just as mucg for clarity as for speed (I wrote it), but how?

----- Original Message ----- 
From: "Raimo Niskanen" <raimo.niskanen@REDACTED>
To: <erlang-questions@REDACTED>
Sent: Monday, May 05, 2003 1:24 PM
Subject: Re: Fix for A=<<1>>


> Alright, fixed for any number of '<' characters. It was a rather simple 
> change. There should be no bad consequences for neither execution time 
> nor memory consumption either, it was just to count the '<' characters 
> and generate the tokens at the end of the sequence.
> 
> If the change breaks anything in our daily build and test runs I will 
> let you know. Otherwise the change will come in R9C.
> 
> / Raimo Niskanen, Erlang/OTP, Ericsson AB
> 
> 
> 
> Ulf Wiger wrote:
> > On Mon, 5 May 2003, Raimo Niskanen wrote:
> > 
> > 
> >>But since sub-binaries are allowed when constructing: how
> >>about '=<<<<<' as in "if A =< << <<1>>/binary, 2>>", then
> >>one can see that the scanning of '=<<' depends on if the
> >>number of '<' characters following is odd or even, so the
> >>scanner might have to scan infinitely ahead. A look ahead
> >>scan of limited small length would be fine, but this is
> >>ugly.
> > 
> > 
> > I'm not sure what the upper limit would be for a sequence of
> > '<' symbols in a program making any qlaims of still being
> > useable (one of course has to take into account generated
> > code, which is usually less readable than hand-written
> > code.)
> > 
> > Perhaps a stupid question, but, so what if the scanner looks
> > ahead and breaks for safety at, say, 1000 tokens? This won't
> > cause any big problems as far as memory is concerned, and at
> > least I find it difficult to envision a program that would
> > break because of this, that is still worthy of being
> > compiled.
> > 
> > The following syntactically correct expression would no
> > longer work (line breaks added for nettiquette compliance).
> > I'm prepared to say "so what?":
> > 
> > A =<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<1>>
> > /binary,2>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>/binary>>/binary>>/binary>>/binary>>/binary>>
> > /binary>>.
> > 
> > /Uffe
> 
>