AFLEX(1)

NAME
aflex - fast lexical analyzer generator for Ada

SYNOPSIS
aflex [ -bdfimpstvEILPTR -Sskeleton_file ] [ filename ]

DESCRIPTION
aflex is a version of the Unix tool lex , but it is written in Ada and generates scanners in Ada. It is upwardly compatible with the UCI tool alex, but is much faster and generates smaller scanners.

OPTIONS
Command line options are given in a different format than in the old UCI alex. Aflex options are as follows

-t  Write the scanner output to the standard output rather than to a file. The default name of the scanner file for base.l is base.a Note that this option is not as useful with aflex because in addition to the scanner file there are files for the externally visible dfa functions (base_dfa.a) and the external IO functions (base_io.a)

-b  Generate backtracking information to aflex.backtrack. This is a list of scanner states which require backtracking and the input characters on which they do so. By adding rules one can remove backtracking states. If all backtracking states are eliminated and -f is used, the generated scanner will run faster (see the -p flag). Only users who wish to squeeze every last cycle out of their scanners need worry about this option.

-d  makes the generated scanner run in debug mode. Whenever a pattern is recognized the scanner will write to stderr a line of the form:
        --accepting rule #n
    Rules are numbered sequentially with the first one being 1. Rule #0 is executed when the scanner backtracks; Rule #(n+1) (where n is the number of rules) indicates the default action; Rule #(n+2) indicates that the input buffer is empty and needs to be refilled and then the scan restarted. Rules beyond (n+2) are end-of-file actions.

-f  has the same effect as lex's -f flag (do not compress the scanner tables); the mnemonic changes from fast compilation to (take your pick) full table or fast scanner. The actual compilation takes longer, since aflex is I/O bound writing out the big table. The compilation of the Ada file containing the scanner is also likely to take a long time because of the large arrays generated.

-i  instructs aflex to generate a case-insensitive scanner. The case of letters given in the aflex input patterns will be ignored, and the rules will be matched regardless of case. The matched text given in yytext will have the preserved case (i.e., it will not be folded).

-m  minimalist with clause. When the -m option is passed, the Ada.Text_IO package, the DFA and IO packages are not with'ed from the main generated lexer.

-p  generates a performance report to stderr. The report consists of comments regarding features of the aflex input file which will cause a loss of performance in the resulting scanner. Note that the use of the ^ operator and the -I flag entail minor performance penalties.

-s  causes the default rule (that unmatched scanner input is echoed to stdout) to be suppressed. If the scanner encounters input that does not match any of its rules, it aborts with an error. This option is useful for finding holes in a scanner's rule set.

-v  has the same meaning as for lex (print to stderr a summary of statistics of the generated scanner). Many more statistics are printed, though, and the summary spans several lines. Most of the statistics are meaningless to the casual aflex user, but the first line identifies the version of aflex, which is useful for figuring out where you stand with respect to patches and new releases.

-E  instructs aflex to generate additional information about each token, including line and column numbers. This is needed for the advanced automatic error option correction in ayacc.

-I  instructs aflex to generate an interactive scanner. Normally, scanners generated by aflex always look ahead one character before deciding that a rule has been matched. At the cost of some scanning overhead, aflex will generate a scanner which only looks ahead when needed. Such scanners are called interactive because if you want to write a scanner for an interactive system such as a command shell, you will probably want the user's input to be terminated with a newline, and without -I the user will have to type a character in addition to the newline in order to have the newline recognized. This leads to dreadful interactive performance.

    If all this seems to confusing, here's the general rule: if a human will be typing in input to your scanner, use -I, otherwise don't; if you don't care about how fast your scanners run and don't want to make any assumptions about the input to your scanner, always use -I.

    Note, -I cannot be used in conjunction with full i.e., the -f flag.

-L  instructs aflex to not generate #line directives (see below).

-P  instructs aflex to generate a private Ada package for the DFA and IO packages. This option is useful to prevent access to the lexer package variables and can be used when the main scanner package is also declared as a private Ada package in the input source file.

-R  generate a reentrant Ada scanner.

-T  makes aflex run in trace mode. It will generate a lot of messages to stdout concerning the form of the input and the resultant non-deterministic and deterministic finite automatons. This option is mostly for use in maintaining aflex.

-Sskeleton_file
    overrides the default internal skeleton from which aflex constructs its scanners. You'll probably never need this option unless you are doing aflex maintenance or development.

SCANNER OPTIONS
aflex supports the flex option directive in the input file but with only one name per line. The syntax is
    %option name

The following options are supported:

case-sensitive or casefull
    Generate a case sensitive scanner (this is the default).

case-insensitive or caseless
    Generate a case insensitive scanner (similar to -i option)

interactive
    Generate an interactive scanner (similar to -I option).

full

yylineno
    Enable the generaton of yylineno and yylinecol variables in the scanner to track the symbol current line and column. The type of the yylineno variable can be controlled by using the yylinenotype option.

nounput or unput
    Disable or enable the generation of the Unput and yyunput function in the _IO package. If you don't use the yyunput no Unput procedure, you should be able to use the nounput option.

noinput or input
    Enable or disable the generation of the Input function in the _IO package.

nooutput or output
    Enable or disable the generation of the Output support in the _IO package.

noyywrap or yywrap
    Enable or disable the use and generation of the yywrap procedure to switch to another buffer or file when the end of a file is reached.

debug
    Enable the debug mode in the scanner (similar to the -d option).

reentrant
    Generate a reentrant scanner (similar to the -R option). When a reentrant scanner is generated, two limited Ada type record are generated in the _DFA and _IO packages to keep track of the current scanner state and the generated scanner does not use global variables. The YYLex function must be called with a context parameter with the type Context_Type defined in the _IO package. The name of context variable can be configured by using the %yyvar directive.

bufsize=NNN
    Controls the size of the read buffer used by the scanner. The default value 75000 has been increased over time to handle large content in the YYText variable. This option allows to control the buffer size.

yylinenotype=type
    Allows to use a specific type for the declaration of the yylineno variable to record line number. The default type is Natural and you can change it to another Ada integer type as long as the initial value is 0. You can use the %yydfa code block to declare such type in the _DFA package.

For the reentrant scanner, two specific directives are added to control the YYLex function declaration and the name of variable that represents the reentrant context.
       %yyname content

yydecl
    This name controls the declaration section of the YYLex function body. The reentrant scanner assumes that the YYLex function has at least one parameter of type Lexer_IO.Context_Type that describes the scanner context. This context will be passed to some generated operation.

yyvar
    This name controls the name of variable representing the scanner context passed to the YYLex function. The default name is Context.

SCANNER CODE BLOCKS
aflex supports code block injection in the generated scanner at various places. This code injection is similar to the code injection provided by flex(1) but dedicated to Ada. The code block can appear in the declaration section and uses the following syntax:
       %yyname {
          -- Put your Ada code here
       }

The following code blocks are supported:

yytype
    This code block is injected in the declaration section of the YYLex function body. It allows you to declare variables, types, local function and procedues that are available to the scanner rules.

yydfa
    This code block is injected in the declaration section of the _DFA package specification. It allows you to declare variables, types, local function and procedues that are available to the _DFA package. A typical use is when you need to use a specific type for the yylineno variable.

yyinit
    This code block shall contain Ada statements which are executed the first time the YYLex function is called. It can be used to setup an initial state for the scanner.

yyaction
    When this code block is defined, aflex will generate the YY_USER_ACTION procedure that is called before executing each action. It should contain Ada statements that are executed in the body of the generated procedure.

yywrap
    When defined, this describes the Ada statements which are used for the generated yywrap function. The Ada statements must return a boolean value. The yywrap function is called when an end-of-file is reached to decide whether the scanner must stop or continue to scan with a next file. The Ada statements shall return true when the scanner must stop and false to proceed with the next input. When this code block is not defined, the default yywrap function will return true .

INCOMPATIBILITIES WITH LEX
aflex is fully compatible with lex with the following exceptions:

•   Source file format:

    The input specification file for aflex must use the following format.
     definitions section
     %%
     rules section
     %%
     user defined section
     ##
     user defined section

•   lex's %r (Ratfor scanners) and %t (translation table) options are not supported.

•   The do-nothing -n flag is not supported.

•   When definitions are expanded, aflex encloses them in parentheses. With lex, the following
        NAME [A-Z][A-Z0-9]*
        %%
        foo{NAME}? Ada.Text_IO.Put_Line ("Found it");
        %%
will not match the string "foo" because when the macro is expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?" and the precedence is such that the '?' is associated with "[A-Z0-9]*". With aflex, the rule will be expanded to "foo([A-z][A-Z0-9]*)?" and so the string "foo" will match. Note that because of this, the ^, $, <s>, and / operators cannot be used in a definition.

•   Input can be controlled by redefining the YY_INPUT function. YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its action is to place up to max_size characters in the character buffer "buf" and return in the integer variable "result" either the number of characters read or the constant YY_NULL to indicate EOF. The default YY_INPUT reads from Standard_Input.

    You also can add in things like counting keeping track of the input line number this way; but don't expect your scanner to go very fast.

•   Yytext is a function returning a String.

•   aflex reads only one input file, while lex's input is made up of the concatenation of its input files.

•   The %unit directive is an optional statement to define the name of the generated Ada package.

•   The following lex constructs are not supported
    - REJECT

    - %T     -- character set tables

    - %x    -- changes to internal array sizes (see below)

ENHANCEMENTS

•   Exclusive start-conditions can be declared by using %x instead of %s. These start-conditions have the property that when they are active, no other rules are active. Thus a set of rules governed by the same exclusive start condition describe a scanner which is independent of any of the other rules in the aflex input. This feature makes it easy to specify "mini-scanners" which scan portions of the input that are syntactically different from the rest (e.g., comments). End-of-file rules. The special rule "<<EOF>>" indicates actions which are to be taken when an end-of-file is encountered and yywrap returns non-zero (i.e., indicates no further files to process). The action can either Ada.Text_IO.Set_Input to a new file to process, in which case the action should finish with YY_NEW_FILE (this is a branch, so subsequent code in the action won't be executed), or it should finish with a return statement. <<EOF>> rules may not be used with other patterns; they may only be qualified with a list of start conditions. If an unqualified <<EOF>> rule is given, it applies only to the INITIAL start condition, and not to %s start conditions. These rules are useful for catching things like unclosed comments. An example:
        %x quote
        %%
        ...
        <quote><<EOF>> {
             error( "unterminated quote" );
             }
        <<EOF>> {
             set_input( next_file );
             YY_NEW_FILE;
             }

•   aflex dynamically resizes its internal tables, so directives like "%a 3000" are not needed when specifying large scanners.

•   aflex generates --#line comments mapping lines in the output to their origin in the input file.

•   All actions must be enclosed by curly braces.

•   Comments may be put in the first section of the input by preceding them with '#'.

•   Ada style comments are supported instead of C style comments.

•   All template files are internalized. The reentrant scanner uses specific templates.

•   The input source file must end with a ".l" extension.

•   The Ada package name used for the generated DFA and IO files can be customized by using the %unit statement at begining of the lex file. By default, aflex will use the scanner file name to build the package name. The package name will be the scanner file name with the possible '-' changed into '.'. The %unit allows to override this and specify the Ada package name to use.

FILES

filename_io.ads

filename_io.adb

filename_dfa.ads

filename_dfa.adb
    The names of the files containing the generated scanner, IO, and DFA packages are based on the basename of the input file. For example if the input file is called scan.l then the scanner file is called scan.ada, the DFA package is in scan_dfa.ads, and scan_io.ads is the IO package file. All of these file names may be changed by modifying the external_file_manager package (see the porting notes for more information.)

aflex.backtrack
    backtracking information for -b

SEE ALSO
lex(1) ayacc(1)

M. E. Lesk and E. Schmidt, LEX - Lexical Analyzer Generator. Technical Report Computing Science Technical Report, 39, Bell Telephone Laboratories, Murray Hill, NJ, 1975.

Military Standard Ada Programming Language
    (ANSI/MIL-STD-1815A-1983), American National Standards Institute, January 1983.

T. Nguyen and K. Forester, Alex - An Ada Lexical Analysis Generator Arcadia Document UCI-88-17, University of California, Irvine, 1988

D. Taback and D. Tolani, Ayacc User's Manual, Arcadia Document UCI-85-10, University of California, Irvine, 1986

AUTHOR
John Self. Based on the tool flex written and designed by Vern Paxson. It reimplements the functionality of the tool alex designed by Thieu Q. Nguyen.

DIAGNOSTICS

aflex scanner jammed - a scanner compiled with -s has encountered an input string which wasn't matched by any of its rules.

old-style lex command ignored - the aflex input contains a lex command (e.g., "%n 1000") which is being ignored.

BUGS

Some trailing context patterns cannot be properly matched and generate warning messages ("Dangerous trailing context"). These are patterns where the ending of the first part of the rule matches the beginning of the second part, such as "zx*/xy*", where the 'x*' matches the 'x' at the beginning of the trailing context. (Lex doesn't get these patterns right either.)

variable trailing context (where both the leading and trailing parts do not have a fixed length) entails a substantial performance loss.

For some trailing context rules, parts which are actually fixed-length are not recognized as such, leading to the abovementioned performance loss. In particular, parts using '|' or {n} are always considered variable-length.

Nulls are not allowed in aflex inputs or in the inputs to scanners generated by aflex. Their presence generates fatal errors.

Pushing back definitions enclosed in ()'s can result in nasty, difficult-to-understand problems like:
        {DIG} [0-9] -- a digit
In which the pushed-back text is "([0-9] -- a digit)".

Due to both buffering of input and read-ahead, you cannot intermix calls to Ada.Text_IO routines, such as, for example, Ada.Text_IO.Get with aflex rules and expect it to work. Call input instead.

There are still more features that could be implemented (especially REJECT) Also the speed of the compressed scanners could be improved.

The utility needs more complete documentation.