4.4BSD/usr/share/man/cat1/lex.0
LEX(1) BSD Reference Manual LEX(1)
NNAAMMEE
lleexx - fast lexical analyzer generator
SSYYNNOOPPSSIISS
lleexx [[--bbccddffiinnppssttvvFFIILLTT88] --CC[eeffmmFF] --SS_s_k_e_l_e_t_o_n] [_f_i_l_e _._._.]
DDEESSCCRRIIPPTTIIOONN
LLeexx is a tool for generating _s_c_a_n_n_e_r_s: programs which recognized lexical
patterns in text. LLeexx reads the given input files, or its standard input
if no file names are given, for a description of a scanner to generate.
The description is in the form of pairs of regular expressions and C
code, called _r_u_l_e_s. LLeexx generates as output a C source file, _l_e_x_._y_y_._c,
which defines a routine yyyylleexx(). This file is compiled and linked with
the --llllnn library to produce an executable. When the executable is run,
it analyzes its input for occurrences of the regular expressions. When-
ever it finds one, it executes the corresponding C code.
For full documentation, see _L_e_x_d_o_c. This manual entry is intended for use
as a quick reference.
OOPPTTIIOONNSS
LLeexx has the following options:
--bb Generate backtracking information to _l_e_x_._b_a_c_k_t_r_a_c_k. This is a
list of scanner states which require backtracking and the input
characters on which they do so. By adding rules one can remove
backtracking states. If all backtracking states are eliminated
and --ff or --FF is used, the generated scanner will run faster.
--cc is a do-nothing, deprecated option included for POSIX compliance.
_N_O_T_E: in previous releases of LLeexx [--cc] specified table-
compression options. This functionality is now given by the --CC
flag. To ease the the impact of this change, when lleexx encounters
--cc,, it currently issues a warning message and assumes that --CC was
desired instead. In the future this "promotion" of --cc to --CC will
go away in the name of full POSIX compliance (unless the POSIX
meaning is removed first).
--dd Makes the generated scanner run in _d_e_b_u_g mode. Whenever a pat-
tern is recognized and the global _y_y___L_e_x___d_e_b_u_g is non-zero (which
is the default), the scanner will write to stderr a line of the
form:
--accepting rule at line 53 ("the matched text")
The line number refers to the location of the rule in the file
defining the scanner (i.e., the file that was fed to lex). Mes-
sages are also generated when the scanner backtracks, accepts the
default rule, reaches the end of its input buffer (or encounters
a NUL; the two look the same as far as the scanner's concerned),
or reaches an end-of-file.
--ff Specifies (take your pick) _f_u_l_l _t_a_b_l_e or _f_a_s_t _s_c_a_n_n_e_r. No table
compression is done. The result is large but fast. This option
is equivalent to --CCff (see below).
--ii Instructs lleexx to generate a _c_a_s_e_-_i_n_s_e_n_s_i_t_i_v_e scanner. The case
of letters given in the lleexx input patterns will be ignored, and
tokens in the input will be matched regardless of case. The
matched text given in _y_y_t_e_x_t will have the preserved case (i.e.,
it will not be folded).
--nn Is another do-nothing, deprecated option included only for POSIX
compliance.
--pp Generates a performance report to stderr. The report consists of
comments regarding features of the lleexx input file which will
cause a loss of performance in the resulting scanner.
--ss Causes the _d_e_f_a_u_l_t _r_u_l_e (that unmatched scanner input is echoed
to _s_t_d_o_u_t) to be suppressed. If the scanner encounters input
that does not match any of its rules, it aborts with an error.
--tt Instructs lleexx to write the scanner it generates to standard out-
put instead of _l_e_x_._y_y_._c.
--vv Specifies that lleexx should write to stderr a summary of statistics
regarding the scanner it generates.
--FF Specifies that the _f_a_s_t scanner table representation should be
used. This representation is about as fast as the full table
representation (--ff), and for some sets of patterns will be con-
siderably smaller (and for others, larger). See _L_e_x_d_o_c for de-
tails.
This option is equivalent to --CCFF (see below).
--II Instructs lleexx to generate an _i_n_t_e_r_a_c_t_i_v_e scanner, that is, a
scanner which stops immediately rather than looking ahead if it
knows that the currently scanned text cannot be part of a longer
rule's match. Again, see _L_e_x_d_o_c for details.
Note, --II cannot be used in conjunction with _f_u_l_l or _f_a_s_t _t_a_b_l_e_s,
i.e., the --ff, --FF, --CCff, or --CCFF flags.
--LL Instructs lleexx not to generate #line directives in _l_e_x_._y_y_._c. The
default is to generate such directives so error messages in the
actions will be correctly located with respect to the original
lleexx input file, and not to the fairly meaningless line numbers of
_l_e_x_._y_y_._c.
--TT Makes lleexx run in _t_r_a_c_e mode. It will generate a lot of messages
to stdout concerning the form of the input and the resultant non-
deterministic and deterministic finite automata. This option is
mostly for use in maintaining lleexx.
--88 Instructs lleexx to generate an 8-bit scanner. On some sites, this
is the default. On others, the default is 7-bit characters. To
see which is the case, check the verbose (--vv) output for "equiva-
lence classes created". If the denominator of the number shown
is 128, then by default lleexx is generating 7-bit characters. If
it is 256, then the default is 8-bit characters.
--CC[eeffmmFF]
Controls the degree of table compression. The default setting is
--CCeemm.
--CC A lone --CC specifies that the scanner tables should be
compressed but neither equivalence classes nor meta-
equivalence classes should be used.
--CCee Directs lleexx to construct _e_q_u_i_v_a_l_e_n_c_e _c_l_a_s_s_e_s, i.e., sets
of characters which have identical lexical properties.
Equivalence classes usually give dramatic reductions in
the final table/object file sizes (typically a factor of
2-5) and are pretty cheap performance-wise (one array
look-up per character scanned).
--CCff Specifies that the _f_u_l_l scanner tables should be generat-
ed - lleexx should not compress the tables by taking advan-
tages of similar transition functions for different
states.
--CCFF Specifies that the alternate fast scanner representation
(described in _L_e_x_d_o_c) should be used.
--CCmm Directs lleexx to construct _m_e_t_a_-_e_q_u_i_v_a_l_e_n_c_e _c_l_a_s_s_e_s, which
are sets of equivalence classes (or characters, if equiv-
alence classes are not being used) that are commonly used
together. Meta-equivalence classes are often a big win
when using compressed tables, but they have a moderate
performance impact (one or two "if" tests and one array
look-up per character scanned).
--CCeemm (Default) Generate both equivalence classes and meta-
equivalence classes. This setting provides the highest
degree of table compression.
Faster-executing scanners can be traded off at the cost of larger
tables with the following generally being true:
slowest & smallest
-Cem
-Cm
-Ce
-C
-C{f,F}e
-C{f,F} fastest & largest
--CC options are not cumulative; whenever the flag is encountered,
the previous -C settings are forgotten.
The options --CCff or --CCFF and --CCmm do not make sense together - there
is no opportunity for meta-equivalence classes if the table is
not being compressed. Otherwise the options may be freely mixed.
--SS_s_k_e_l_e_t_o_n___f_i_l_e
Overrides the default skeleton file from which lleexx constructs its
scanners. Useful for lleexx maintenance or development.
SSUUMMMMAARRYY OOFF LLEEXX RREEGGUULLAARR EEXXPPRREESSSSIIOONNSS
The patterns in the input are written using an extended set of regular
expressions. These are:
x Match the character 'x'.
. Any character except newline.
[xyz] A "character class"; in this case, the pattern matches either
an 'x', a 'y', or a 'z'.
[abj-oZ] A "character class" with a range in it; matches an 'a', a
'b', any letter from 'j' through 'o', or a 'Z'.
[^A-Z] A "negated character class", i.e., any character but those in
the class. In this case, any character _e_x_c_e_p_t an uppercase
letter.
[^A-Z\n] Any character _e_x_c_e_p_t an uppercase letter or a newline.
r* Zero or more r's, where r is any regular expression.
r+ One or more r's.
r? Zero or one r's (that is, "an optional r").
r{2,5} Anywhere from two to five r's.
r{2,} Two or more r's.
r{4} Exactly 4 r's.
{name} The expansion of the "name" definition (see above).
[xyz]\"foo The literal string: [xyz]"foo.
\X If X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v', then the
ANSI-C interpretation of \x. Otherwise, a literal 'X' (used
to escape operators such as '*').
\123 The character with octal value 123.
\x2a The character with hexadecimal value 2a.
(r) Match an r; parentheses are used to override precedence (see
below).
rs The regular expression r followed by the regular expression
s; called "concatenation".
rs Either an r or an s.
r/s An r but only if it is followed by an s. The s is not part
of the matched text. This type of pattern is called as
"trailing context".
^r An r, but only at the beginning of a line.
r$ An r, but only at the end of a line. Equivalent to "r/\n".
<s>r An r, but only in start condition s (see below for discussion
of start conditions).
<s1,s2,s3>r
Same, but in any of start conditions s1, s2, or s3.
<<EOF>> An end-of-file.
<s1,s2><<EOF>>
An end-of-file when in start condition s1 or s2.
The regular expressions listed above are grouped according to precedence,
from highest precedence at the top to lowest at the bottom. Those
grouped together have equal precedence.
Some notes on patterns:
Negated character classes _m_a_t_c_h _n_e_w_l_i_n_e_s unless "\n" (or an equivalent
escape sequence) is one of the characters explicitly present in the
negated character class (e.g., " [^A-Z\n] ").
A rule can have at most one instance of trailing context (the '/' opera-
tor or the '$' operator). The start condition, '^', and "<<EOF>>" pat-
terns can only occur at the beginning of a pattern, and, as well as with
'/' and '$', cannot be grouped inside parentheses. The following are all
illegal:
foo/bar$
foo(bar$)
foo^bar
<sc1>foo<sc2>bar
SSUUMMMMAARRYY OOFF SSPPEECCIIAALL AACCTTIIOONNSS
In addition to arbitrary C code, the following can appear in actions:
EECCHHOO Copies _y_y_t_e_x_t to the scanner's output.
BBEEGGIINN Followed by the name of a start condition places the scanner
in the corresponding start condition.
RREEJJEECCTT Directs the scanner to proceed on to the "second best" rule
which matched the input (or a prefix of the input). _y_y_t_e_x_t
and _y_y_l_e_n_g are set up appropriately. Note that RREEJJEECCTT is a
particularly expensive feature in terms scanner performance;
if it is used in _a_n_y of the scanner's actions it will slow
down _a_l_l of the scanner's matching. Furthermore, RREEJJEECCTT can-
not be used with the --ff or --FF options.
Note also that unlike the other special actions, RREEJJEECCTT is a
_b_r_a_n_c_h; code immediately following it in the action will _n_o_t
be executed.
yyyymmoorree() tells the scanner that the next time it matches a rule, the
corresponding token should be _a_p_p_e_n_d_e_d onto the current value
of _y_y_t_e_x_t rather than replacing it.
yyyylleessss(_n) returns all but the first _n characters of the current token
back to the input stream, where they will be rescanned when
the scanner looks for the next match. _y_y_t_e_x_t and _y_y_l_e_n_g are
adjusted appropriately (e.g., _y_y_l_e_n_g will now be equal to _n).
uunnppuutt(_c) puts the character _c back onto the input stream. It will be
the next character scanned.
iinnppuutt() reads the next character from the input stream (this routine
is called yyyyiinnppuutt() if the scanner is compiled using _C _+_+).
yyyytteerrmmiinnaattee()
can be used in lieu of a return statement in an action. It
terminates the scanner and returns a 0 to the scanner's
caller, indicating "all done".
By default, yyyytteerrmmiinnaattee() is also called when an end-of-file
is encountered. It is a macro and may be redefined.
YYYY__NNEEWW__FFIILLEE
is an action available only in <<EOF>> rules. It means
"Okay, I've set up a new input file, continue scanning".
yyyy__ccrreeaattee__bbuuffffeerr(_f_i_l_e, _s_i_z_e)
takes a FFIILLEE pointer and an integer _s_i_z_e. It returns a
YY_BUFFER_STATE handle to a new input buffer large enough to
accomodate _s_i_z_e characters and associated with the given
file. When in doubt, use _Y_Y___B_U_F___S_I_Z_E for the size.
yyyy__sswwiittcchh__ttoo__bbuuffffeerr(_n_e_w___b_u_f_f_e_r)
switches the scanner's processing to scan for tokens from the
given buffer, which must be a YY_BUFFER_STATE.
yyyy__ddeelleettee__bbuuffffeerr(_b_u_f_f_e_r)
deletes the given buffer.
VVAALLUUEESS AAVVAAIILLAABBLLEE TTOO TTHHEE UUSSEERR
_c_h_a_r _*_y_y_t_e_x_t
holds the text of the current token. It may not be modified.
_i_n_t _y_y_l_e_n_g holds the length of the current token. It may not be modi-
fied.
_F_I_L_E _*_y_y_i_n is the file which by default lleexx reads from. It may be rede-
fined but doing so only makes sense before scanning begins.
Changing it in the middle of scanning will have unexpected
results since lleexx buffers its input. Once scanning termi-
nates because an end-of-file has been seen, vvooiidd
yyyyrreessttaarrtt(_F_I_L_E _*_n_e_w___f_i_l_e) may be called to point _y_y_i_n at the
new input file.
_F_I_L_E _*_y_y_o_u_t
is the file to which _E_C_H_O actions are done. It can be reas-
signed by the user.
_Y_Y___C_U_R_R_E_N_T___B_U_F_F_E_R
returns a YY_BUFFER_STATE handle to the current buffer.
MMAACCRROOSS TTHHEE UUSSEERR CCAANN RREEDDEEFFIINNEE
_Y_Y___D_E_C_L controls how the scanning routine is declared. By default,
it is "int yylex()", or, if prototypes are being used, "int
yylex(void)". This definition may be changed by redefining
the "YY_DECL" macro. Note that if you give arguments to the
scanning routine using a K&R-style/non-prototyped function
declaration, you must terminate the definition with a semi-
colon (;).
_Y_Y___I_N_P_U_T The nature of how the scanner gets its input can be con-
trolled by redefining the YY_INPUT macro. YY_INPUT's calling
sequence is "YY_INPUT(buf,result,max_size)". Its action is
to place up to _m_a_x ___s_i_z_e characters in the character array
_b_u_f and return in the integer variable _r_e_s_u_l_t either the num-
ber of characters read or the constant YY_NULL (0 on Unix
systems) to indicate EOF. The default YY_INPUT reads from
the global file-pointer "yyin". A sample redefinition of
YY_INPUT (in the definitions section of the input file):
%{
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) \
result = ((buf[0] = getchar()) == EOF) ? YY_NULL : 1;
%}
_Y_Y___I_N_P_U_T When the scanner receives an end-of-file indication from
YY_INPUT, it then checks the yyyywwrraapp() function. If yyyywwrraapp()
returns false (zero), then it is assumed that the function
has gone ahead and set up _y_y_i_n to point to another input
file, and scanning continues. If it returns true (non-zero),
then the scanner terminates, returning 0 to its caller.
_y_y_w_r_a_p The default yyyywwrraapp() always returns 1.
_Y_Y___U_S_E_R___A_C_T_I_O_N
can be redefined to provide an action which is always execut-
ed prior to the matched rule's action.
_Y_Y___U_S_E_R___I_N_I_T
The macro _Y_Y ___U_S_E_R___I_N_I_T may be redefined to provide an action
which is always executed before the first scan.
_Y_Y___B_R_E_A_K In the generated scanner, the actions are all gathered in one
large switch statement and separated using _Y_Y ___B_R_E_A_K, which
may be redefined. By default, it is simply a "break", to
separate each rule's action from the following rule's.
FFIILLEESS
/usr/lib/libln.a object code library.
lex.skel skeleton scanner.
lex.yy.c generated scanner (called _l_e_x_y_y_._c on some systems).
lex.backtrack backtracking information for --bb _f_l_a_g (called _l_e_x_._b_c_k on
some systems).
SSEEEE AALLSSOO
yacc(1), sed(1), awk(1).
_l_e_x_d_o_c.
M. E. Lesk, and E. Schmidt, _L_E_X _- _L_e_x_i_c_a_l _A_n_a_l_y_z_e_r _G_e_n_e_r_a_t_o_r.
DDIIAAGGNNOOSSTTIICCSS
reject_used_but_not_detected undefined
or
yymore_used_but_not_detected undefined
These errors can occur at compile time. They indicate that
the scanner uses RREEJJEECCTT or yyyymmoorree() but that lleexx failed to
notice the fact, meaning that lleexx scanned the first two sec-
tions looking for occurrences of these actions and failed to
find any, but somehow you snuck some in via a #include file,
for example . Make an explicit reference to the action in
your lleexx input file. Note that previously lleexx supported a
%used/%unused mechanism for dealing with this problem; this
feature is still supported but now deprecated, and will go
away soon unless the author hears from people who can argue
compellingly that they need it.
lex scanner jammed
a scanner compiled with --ss has encountered an input string
which wasn't matched by any of its rules.
lex input buffer overflowed
a scanner rule matched a string long enough to overflow the
scanner's internal input buffer 16K bytes - controlled by
_Y_Y___B_U_F___M_A_X in _l_e_x_._s_k_e_l.
scanner requires -8 flag
Your scanner specification includes recognizing 8-bit charac-
ters and you did not specify the -8 flag and your site has
not installed lex with -8 as the default .
too many %t classes!
You managed to put every single character into its own %t
class. LLeexx requires that at least one of the classes share
characters.
CCOOMMPPAATTIIBBIILLIITTYY
The undocumented lleexx scanner internal variable _y_y_l_i_n_e_n_o is not supported
in this implementation. It is difficult to support this option effi-
ciently, since it requires examining every character scanned and reexam-
ining the characters when the scanner backs up. Things get more compli-
cated when the end of buffer or file is reached or a NUL is scanned
(since the scan must then be restarted with the proper line number
count), or the user uses the yyyylleessss(), uunnppuutt(), or REJECT actions, or the
multiple input buffer functions.
The workaround is to add rules which, upon seeing a newline, increment
_y_y_l_i_n_e_n_o. This is usually an easy process, though it can be difficult if
any of the patterns can match multiple newlines along with other charac-
ters.
The _y_y_l_i_n_e_n_o variable is not specified by the IEEE Std1003.2 (``POSIX'')
specification.
HHIISSTTOORRYY
A lleexx appeared in Version 6 AT&T UNIX. The version this man page de-
scribes is derived from code contributed by Vern Paxson.
AAUUTTHHOORR
Vern Paxson, with the help of many ideas and much inspiration from Van
Jacobson. Original version by Jef Poskanzer.
See _L_e_x_d_o_c for additional credits and the address to send comments to.
BBUUGGSS
Some trailing context patterns cannot be properly matched and generate
warning messages ("Dangerous trailing context"). These are patterns
where the ending of the first part of the rule matches the beginning of
the second part, such as "zx*/xy*", where the 'x*' matches the 'x' at the
beginning of the trailing context. (Note that the IEEE Std1003.2
(``POSIX'') specification states that the text matched by such patterns
is undefined.)
For some trailing context rules, parts which are actually fixed-length
are not recognized as such, leading to the abovementioned performance
loss. In particular, parts using '|' or {n} (such as "foo{3}") are al-
ways considered variable-length.
Combining trailing context with the special '|' action can result in
_f_i_x_e_d trailing context being turned into the more expensive _v_a_r_i_a_b_l_e
trailing context. This happens in the following example:
%%
abc |
xyz/def
Use of uunnppuutt() invalidates yytext and yyleng.
Use of uunnppuutt() to push back more text than was matched can result in the
pushed-back text matching a beginning-of-line ('^') rule even though it
didn't come at the beginning of the line (though this is rare!).
Pattern-matching of NUL's is substantially slower than matching other
characters.
LLeexx does not generate correct #line directives for code internal to the
scanner; thus, bugs in _l_e_x_._s_k_e_l yield bogus line numbers.
Due to both buffering of input and read-ahead, you cannot intermix calls
to <_s_t_d_i_o_._h> routines, such as, for example, ggeettcchhaarr(), with lleexx rules
and expect it to work. Call iinnppuutt() instead.
The total table entries listed by the --vv flag excludes the number of
table entries needed to determine what rule has been matched. The number
of entries is equal to the number of DFA states if the scanner does not
use RREEJJEECCTT, and somewhat greater than the number of states if it does.
RREEJJEECCTT cannot be used with the --ff or --FF options.
The lleexx internal algorithms need documentation.
4.4BSD June 6, 1993 8