
## NAME

aflex - fast lexical analyzer generator for Ada

## SYNOPSIS

*aflex* [
*-bdfimpstvEILPTR -Sskeleton_file* ] [ 
*filename* ]

## DESCRIPTION

*aflex* is a version of the Unix tool 
*lex* , but it is written in Ada and generates scanners in Ada.
It is upwardly compatible with the UCI tool alex, but is
much faster and generates smaller scanners.

## OPTIONS

Command line options are given in a different format than in the
old UCI alex.  Aflex options are as follows


*-t* Write the scanner output to the standard output rather than to a file.
The default name of the scanner file for base.l is base.a  Note that this
option is not as useful with aflex because in addition to the scanner
file there are files for the externally visible dfa functions
(base_dfa.a) and the external IO functions (base_io.a)


*-b* Generate backtracking information to
*aflex.backtrack.* This is a list of scanner states which require backtracking
and the input characters on which they do so.  By adding rules one
can remove backtracking states.  If all backtracking states
are eliminated and
*-f* is used, the generated scanner will run faster (see the
*-p* flag).  Only users who wish to squeeze every last cycle out of their
scanners need worry about this option.


*-d* makes the generated scanner run in
*debug* mode.  Whenever a pattern is recognized the scanner will
write to
*stderr* a line of the form:

    --accepting rule #n

Rules are numbered sequentially with the first one being 1.  Rule #0
is executed when the scanner backtracks; Rule #(n+1) (where
*n* is the number of rules) indicates the default action; Rule #(n+2) indicates
that the input buffer is empty and needs to be refilled and then the scan
restarted.  Rules beyond (n+2) are end-of-file actions.


*-f* has the same effect as lex's -f flag (do not compress the scanner
tables); the mnemonic changes from
*fast compilation* to (take your pick)
*full table* or
*fast scanner.* The actual compilation takes
*longer,* since aflex is I/O bound writing out the big table.
The compilation of the Ada file containing the scanner is also likely
to take a long time because of the large arrays generated.


*-i* instructs aflex to generate a
*case-insensitive* scanner.  The case of letters given in the aflex input patterns will
be ignored, and the rules will be matched regardless of case.  The
matched text given in
*yytext* will have the preserved case (i.e., it will not be folded).


*-m* minimalist with clause.  When the
*-m* option is passed, the
*Ada.Text_IO* package, the DFA and IO packages are not with'ed from the main
generated lexer.


*-p* generates a performance report to stderr.  The report
consists of comments regarding features of the aflex input file
which will cause a loss of performance in the resulting scanner.
Note that the use of
the
*^* operator
and the
*-I* flag entail minor performance penalties.


*-s* causes the
*default rule* (that unmatched scanner input is echoed to
*stdout)* to be suppressed.  If the scanner encounters input that does not
match any of its rules, it aborts with an error.  This option is
useful for finding holes in a scanner's rule set.


*-v* has the same meaning as for lex (print to
*stderr* a summary of statistics of the generated scanner).  Many more statistics
are printed, though, and the summary spans several lines.  Most
of the statistics are meaningless to the casual aflex user, but the
first line identifies the version of aflex, which is useful for figuring
out where you stand with respect to patches and new releases.


*-E* instructs aflex to generate additional information about each token,
including line and column numbers.  This is needed for the advanced
automatic error option correction in ayacc.


*-I* instructs aflex to generate an
*interactive* scanner.  Normally, scanners generated by aflex always look ahead one
character before deciding that a rule has been matched.  At the cost of
some scanning overhead, aflex will generate a scanner which only looks ahead
when needed.  Such scanners are called
*interactive* because if you want to write a scanner for an interactive system such as a
command shell, you will probably want the user's input to be terminated
with a newline, and without
*-I* the user will have to type a character in addition to the newline in order
to have the newline recognized.  This leads to dreadful interactive
performance.

    If all this seems to confusing, here's the general rule: if a human will
    be typing in input to your scanner, use
*-I,*     otherwise don't; if you don't care about how fast your scanners run and
    don't want to make any assumptions about the input to your scanner,
    always use
*-I.* 
    Note,
*-I*     cannot be used in conjunction with
*full*     i.e., the
*-f*     flag.


*-L* instructs aflex to not generate
*#line* directives (see below).


*-P* instructs aflex to generate a private Ada package for the DFA and IO packages.
This option is useful to prevent access to the lexer package variables
and can be used when the main scanner package is also declared as a private Ada package
in the input source file.


*-R* generate a reentrant Ada scanner.


*-T* makes aflex run in
*trace* mode.  It will generate a lot of messages to stdout concerning
the form of the input and the resultant non-deterministic and deterministic
finite automatons.  This option is mostly for use in maintaining aflex.


*-Sskeleton_file* overrides the default internal skeleton from which aflex constructs
its scanners.  You'll probably never need this option unless you are doing
aflex maintenance or development.

## SCANNER OPTIONS

*aflex* supports the
*flex* option directive in the input file but with only one name per line.  The syntax is
```

%option name
```

The following options are supported:


case-sensitive or casefull
Generate a case sensitive scanner (this is the default).


case-insensitive or caseless
Generate a case insensitive scanner (similar to
*-i* option)


interactive
Generate an interactive scanner (similar to
*-I* option).


full


yylineno
Enable the generaton of
*yylineno* and
*yylinecol* variables in the scanner to track the symbol current line and column.
The type of the
*yylineno* variable can be controlled by using the
*yylinenotype* option.


nounput or unput
Disable or enable the generation of the
*Unput* and
*yyunput* function in the
*_IO* package.  If you don't use the
*yyunput* no
*Unput* procedure, you should be able to use the
*nounput* option.


noinput or input
Enable or disable the generation of the
*Input* function in the
*_IO* package.


nooutput or output
Enable or disable the generation of the
*Output* support in the
*_IO* package.


noyywrap or yywrap
Enable or disable the use and generation of the
*yywrap* procedure to switch to another buffer or file when the end of a file is reached.


debug
Enable the debug mode in the scanner (similar to the
*-d* option).


reentrant
Generate a reentrant scanner (similar to the
*-R* option).  When a reentrant scanner is generated, two limited
Ada type record are generated in the
*_DFA* and
*_IO* packages to keep track of the current scanner state and the generated scanner
does not use global variables.  The
*YYLex* function must be called with a context parameter with the type
*Context_Type* defined in the
*_IO* package.  The name of context variable can be configured by using the
*%yyvar* directive.


bufsize=NNN
Controls the size of the read buffer used by the scanner.  The default value
*75000* has been increased over time to handle large content in the
*YYText* variable.  This option allows to control the buffer size.


yylinenotype=**type**
Allows to use a specific type for the declaration of the
*yylineno* variable to record line number.  The default type is
*Natural* and you can change it to another Ada integer type as long as the initial
value is 0.  You can use the
*%yydfa* code block to declare such type in the
*_DFA* package.

For the reentrant scanner, two specific directives are added to control the
*YYLex* function declaration and the name of variable that represents the reentrant
context.
```

   %yyname content
```


yydecl
This name controls the declaration section of the
*YYLex* function body.  The reentrant scanner assumes that the
*YYLex* function has at least one parameter of type
*Lexer_IO.Context_Type* that describes the scanner context.  This context will be passed to some generated
operation.


yyvar
This name controls the name of variable representing the scanner context
passed to the
*YYLex* function.  The default name is
**Context**. 
## SCANNER CODE BLOCKS

*aflex* supports code block injection in the generated scanner at various places.
This code injection is similar to the code injection provided by
**flex**(1) but dedicated to Ada.  The code block can appear in the declaration section
and uses the following syntax:
```

   %yyname {
      -- Put your Ada code here
   }
```

The following code blocks are supported:


yytype
This code block is injected in the declaration section of the
*YYLex* function body.  It allows you
to declare variables, types, local function and procedues that are available
to the scanner rules.


yydfa
This code block is injected in the declaration section of the
*_DFA* package specification.  It allows you
to declare variables, types, local function and procedues that are available
to the
*_DFA* package.  A typical use is when you need to use a specific type for the
*yylineno* variable.


yyinit
This code block shall contain Ada statements which are executed the first time the
*YYLex* function is called.  It can be used to setup an initial state for the scanner.


yyaction
When this code block is defined,
*aflex* will generate the
*YY_USER_ACTION* procedure that is called before executing each action.  It should contain
Ada statements that are executed in the body of the generated procedure.


yywrap
When defined, this describes the Ada statements which are used for the generated
*yywrap* function.  The Ada statements must return a boolean value.
The
*yywrap* function is called when an end-of-file is reached to decide whether the scanner must stop
or continue to scan with a next file.  The Ada statements shall return
*true* when the scanner must stop and
*false* to proceed with the next input.  When this code block is not defined, the default
*yywrap* function will return
*true .* 
## INCOMPATIBILITIES WITH LEX

*aflex* is fully compatible with
*lex* with the following exceptions:

* Source file format:
The input specification file for 
*aflex* must use the following format.
```
 definitions section
 %%
 rules section
 %%
 user defined section
 ##
 user defined section
```

* lex's
*%r* (Ratfor scanners) and
*%t* (translation table) options
are not supported.

* The do-nothing
*-n* flag is not supported.

* When definitions are expanded, aflex encloses them in parentheses.
With lex, the following
```

    NAME    [A-Z][A-Z0-9]*
    %%
    foo{NAME}?      Ada.Text_IO.Put_Line ("Found it");
    %%

```
will not match the string "foo" because when the macro
is expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?"
and the precedence is such that the '?' is associated with
"[A-Z0-9]*".  With aflex, the rule will be expanded to
"foo([A-z][A-Z0-9]*)?" and so the string "foo" will match.
Note that because of this, the
^, $, \<s\>, and / operators cannot be used in a definition.

* Input can be controlled by redefining the
*YY_INPUT* function.
YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)".  Its
action is to place up to max_size characters in the character buffer "buf"
and return in the integer variable "result" either the
number of characters read or the constant YY_NULL
to indicate EOF.  The default YY_INPUT reads from
Standard_Input.
You also can add in things like counting keeping track of the
input line number this way; but don't expect your scanner to
go very fast.

* Yytext is a function returning a
*String*. 
* aflex reads only one input file, while lex's input is made
up of the concatenation of its input files.

* The
*%unit* directive is an optional statement to define the name of the generated Ada package.

* The following lex constructs are not supported
```
- REJECT
- %T 	-- character set tables
- %x	-- changes to internal array sizes (see below)
```

## ENHANCEMENTS


* *Exclusive start-conditions* can be declared by using
*%x* instead of
*%s.* These start-conditions have the property that when they are active,
*no other rules are active.* Thus a set of rules governed by the same exclusive start condition
describe a scanner which is independent of any of the other rules in
the aflex input.  This feature makes it easy to specify "mini-scanners"
which scan portions of the input that are syntactically different
from the rest (e.g., comments).
*End-of-file rules.* The special rule "<<EOF>>" indicates
actions which are to be taken when an end-of-file is
encountered and
*yywrap* returns non-zero (i.e., indicates
no further files to process).  The action can either
*Ada.Text_IO.Set_Input* to a new file to process, in which case the
action should finish with
*YY_NEW_FILE* (this is a branch, so subsequent code in the action won't
be executed), or it should finish with a
*return* statement.  <<EOF>> rules may not be used with other
patterns; they may only be qualified with a list of start
conditions.  If an unqualified <<EOF>> rule is given, it
applies only to the INITIAL start condition, and
*not* to
*%s* start conditions.
These rules are useful for catching things like unclosed comments.
An example:
```

    %x quote
    %%
    ...
    <quote><<EOF>>   {
	     error( "unterminated quote" );
	     }
    <<EOF>>          {
	     set_input( next_file );
	     YY_NEW_FILE;
	     }

```

* aflex dynamically resizes its internal tables, so directives like "%a 3000"
are not needed when specifying large scanners.

* aflex generates
*--#line* comments mapping lines in the output to
their origin in the input file.

* All actions must be enclosed by curly braces.

* Comments may be put in the first section of the input by preceding
them with '#'.

* Ada style comments are supported instead of C style comments.

* All template files are internalized.  The reentrant scanner uses specific templates.

* The input source file must end with a ".l" extension.

* The Ada package name used for the generated DFA and IO files can be customized by
using the
*%unit* statement at begining of the lex file.  By default, aflex will use the
scanner file name to build the package name.  The package name will be the
scanner file name with the possible '-' changed into '.'.  The
*%unit* allows to override this and specify the Ada package name to use.

## FILES



**filename**_io.ads 

**filename**_io.adb 

**filename**_dfa.ads 

**filename**_dfa.adb The names of the files containing the generated scanner, IO,
and DFA packages are based on the basename of the input file.
For example if the input file is called scan.l then the
scanner file is called **scan.ada**, the DFA package is in **scan_dfa.ads**, and
**scan_io.ads** is the IO package file.  All of these file names may be changed
by modifying the external_file_manager package (see the porting notes
for more information.)


*aflex.backtrack* backtracking information for
*-b* 
## SEE ALSO

*lex*(1) *ayacc*(1) 
M. E. Lesk and E. Schmidt,
*LEX - Lexical Analyzer Generator.* Technical Report Computing Science Technical Report, 39, Bell Telephone
Laboratories, Murray Hill, NJ, 1975.

Military Standard Ada Programming Language
	(ANSI/MIL-STD-1815A-1983),
American National Standards Institute, January 1983.

T. Nguyen and K. Forester,
*Alex - An Ada Lexical Analysis Generator* Arcadia Document UCI-88-17,
University of California, Irvine, 1988

D. Taback and D. Tolani,
*Ayacc User's Manual,* Arcadia Document UCI-85-10,
University of California, Irvine, 1986

## AUTHOR

John Self.  Based on the tool flex written and designed by
Vern Paxson.  It reimplements the functionality of the tool alex
designed by Thieu Q. Nguyen.

## DIAGNOSTICS


*aflex scanner jammed -* a scanner compiled with
*-s* has encountered an input string which wasn't matched by
any of its rules.


*old-style lex command ignored -* the aflex input contains a lex command (e.g., "%n 1000") which
is being ignored.

## BUGS


Some trailing context
patterns cannot be properly matched and generate
warning messages ("Dangerous trailing context").  These are
patterns where the ending of the
first part of the rule matches the beginning of the second
part, such as "zx*/xy*", where the 'x*' matches the 'x' at
the beginning of the trailing context.  (Lex doesn't get these
patterns right either.)

*variable* trailing context (where both the leading and trailing parts do not have
a fixed length) entails a substantial performance loss.

For some trailing context rules, parts which are actually fixed-length are
not recognized as such, leading to the abovementioned performance loss.
In particular, parts using '|' or {n} are always considered variable-length.

Nulls are not allowed in aflex inputs or in the inputs to
scanners generated by aflex.  Their presence generates fatal
errors.

Pushing back definitions enclosed in ()'s can result in nasty,
difficult-to-understand problems like:
```

	{DIG}  [0-9] -- a digit

```
In which the pushed-back text is "([0-9] -- a digit)".

Due to both buffering of input and read-ahead, you cannot intermix
calls to
*Ada.Text_IO* routines, such as, for example,
*Ada.Text_IO.Get* with aflex rules and expect it to work.  Call
*input* instead.

There are still more features that could be implemented (especially REJECT) 
Also the speed of the compressed scanners could be improved.

The utility needs more complete documentation.
