Changes by: Jason Hickey (jyh at cs.caltech.edu)
Date: 2005-02-23 20:53:46 -0800 (Wed, 23 Feb 2005)
Revision: 6757
Log message:
Added nested lexing to handle nested matching pairs.
The syntax is:
lex_token <lexer> : <left> <right> [--> <token>]
So, a Caml-style nesting comment could be ignored as follows.
lex_token mmc : "[(][*]" "[*][)]"
Quotations are handled this way too.
(*
* arg1 is the start delimiter, arg2 is the end delimiter.
* the default quotation is "term".
*)
lex_token mmc : "<<" ">>" --> tok_quotation{xquotation[arg1:s, lexeme:s]}
The xquotations are expanded eagerly, at lex time. They are still
expanded as iforms, so you can always type them in directly if you want.
iform parse_as_term : foo[q:s] <--> xquotation["term", q:s]