Erlang style comments starting with a % are allowed in
scanner files. A definition file has the following format:
<Header>
Definitions.
<Macro Definitions>
Rules.
<Token Rules>
Erlang code.
<Erlang code>
The "Definitions.", "Rules." and "Erlang code." headings are
mandatory and must occur at the beginning of a source line. The
<Header>, <Macro Definitions> and <Erlang code>
sections may be empty but there must be at least one rule.
Macro definitions have the following format:
and there must be spaces around =. Macros can be used in
the regular expressions of rules by writing {NAME}.
Note
When macros are expanded in expressions the macro calls
are replaced by the macro value without any form of quoting or
enclosing in parentheses.
Rules have the following format:
<Regexp> : <Erlang code>.
The <Regexp> must occur at the start of a line and not
include any blanks; use \t and \s to include TAB
and SPACE characters in the regular expression. If <Regexp>
matches then the corresponding <Erlang code> is evaluated to
generate a token. With the Erlang code the following predefined
variables are available:
- TokenChars
-
A list of the characters in the matched token.
- TokenLen
-
The number of characters in the matched token.
- TokenLine
-
The line number where the token occurred.
- TokenCol
-
The column number where the token occurred
(column of the first character included in the token).
- TokenLoc
-
Token location. Expands to {TokenLine,TokenCol} (even
when error_location is set to line.
The code must return:
- {token,Token}
-
Return Token to the caller.
- {end_token,Token}
-
Return Token and is last token in a tokens call.
- skip_token
-
Skip this token completely.
- {error,ErrString}
-
An error in the token, ErrString is a string
describing the error.
It is also possible to push back characters into the input
characters with the following returns:
- {token,Token,PushBackList}
- {end_token,Token,PushBackList}
- {skip_token,PushBackList}
These have the same meanings as the normal returns but the
characters in PushBackList will be prepended to the input
characters and scanned for the next token. Note that pushing
back a newline will mean the line numbering will no longer be
correct.
Note
Pushing back characters gives you unexpected
possibilities to cause the scanner to loop!
The following example would match a simple Erlang integer or
float and return a token which could be sent to the Erlang
parser:
D = [0-9]
{D}+ :
{token,{integer,TokenLine,list_to_integer(TokenChars)}}.
{D}+\.{D}+((E|e)(\+|\-)?{D}+)? :
{token,{float,TokenLine,list_to_float(TokenChars)}}.
The Erlang code in the "Erlang code." section is written into
the output file directly after the module declaration and
predefined exports declaration so it is possible to add extra
exports, define imports and other attributes which are then
visible in the whole file.