Hi all,
I'm preparing to write a filter that will replace M4 in flex. I plan to write it using flex & bison so we won't gain any additional dependencies.
I think we need a pretty basic macro processor with requirements close to these:
- Processes one line at a time
- Recognizes flex's %keyword style commands, including %% and %# markers
- Processes macro replacements once per line.
-- Nested macros won't work because the line isn't re-scanned after expansion, for example
- Permits user-defined macros that comprise a name and its replacement string.
-- Macro names may contain any alphanumeric character and underscore
-- Macro names may begin with any alphabetic character or underscore
-- Any reason for more/less flexibility? We could just as easily make the names like %$CODEPOINTS
- Macros are immutable after definition
-- This plus refusal to re-scan after expansion close the highway to accidental Turing completeness and make the test suite simpler.
-- I'm not married to this requirement if there's a call to allow re-definition (or if I realize we need it for a skeleton down the line).
- Emits processed lines to stdout
- Loops over lines (e.g. of the skeleton) until reaching a skelout marker, %%, then returns
- Receives a copy of the scanner options at invocation, e.g. to read emission control options - Maybe
- Returns a copy of the scanner options on exit, e.g. to report changes to emission control options - Maybe
-- On this and the Maybe above, I think this would be useful in place of emitting M4_IN_HEADER style variables and redefining them a line at a time as the skeleton is processed. That current system assumes the M4 filter instance downstream of the skeleton's filter will receive inline macros emitted from the skeleton itself and switch header output on or off. We had to write support for that into each skeleton. If we're writing our own filter we can pass a more flexible memento without adding a ton of overhead to each new skeleton. I'm half trying to talk myself into this idea. Please, someone else chime in.
What else? Do we need to build this as a clean room implementation for any reason?
Hi all,
I'm preparing to write a filter that will replace M4 in flex. I plan to write it using flex & bison so we won't gain any additional dependencies.
I think we need a pretty basic macro processor with requirements close to these:
-- Nested macros won't work because the line isn't re-scanned after expansion, for example
-- Macro names may contain any alphanumeric character and underscore
-- Macro names may begin with any alphabetic character or underscore
-- Any reason for more/less flexibility? We could just as easily make the names like %$CODEPOINTS
-- This plus refusal to re-scan after expansion close the highway to accidental Turing completeness and make the test suite simpler.
-- I'm not married to this requirement if there's a call to allow re-definition (or if I realize we need it for a skeleton down the line).
-- On this and the Maybe above, I think this would be useful in place of emitting M4_IN_HEADER style variables and redefining them a line at a time as the skeleton is processed. That current system assumes the M4 filter instance downstream of the skeleton's filter will receive inline macros emitted from the skeleton itself and switch header output on or off. We had to write support for that into each skeleton. If we're writing our own filter we can pass a more flexible memento without adding a ton of overhead to each new skeleton. I'm half trying to talk myself into this idea. Please, someone else chime in.
What else? Do we need to build this as a clean room implementation for any reason?