Parsing Indentet Blocks

[furesoft]

To parse languages like python you have to have a system to parse indented blocks.

Could be implemented as follows:

1. Lexer produces INDENT and DEDENT tokens
2. Special parselet, that parses those

[furesoft]

To recognize indentations generically in the framework (without an explicit colon), the lexer must be extended so that it counts the indentation at the beginning of each line and generates corresponding INDENT/DEDENT tokens. The parser logic must recognize blocks based on these tokens, regardless of any preceding characters.

Procedure:

Lexer:

Count the indentation (spaces/tabs) at the beginning of the line.

Compare with the previous line.

If there is more indentation: INDENT token.

If there is less indentation: one DEDENT token for each level.

If the indentation is the same: no token.

Parser:

Treat INDENT as the start of a block, DEDENT as the end of a block.

No reference to : or other block delimiters.

// Lexer: Recognize indentation in pseudocode
Stack<int> indentStack = new Stack<int>();
indentStack.Push(0);

foreach (Line z in source) {
    int currentIndent = CountIndent(z);
    int lastIndent = indentStack.Peek();

    if (currentIndent > lastIndent) {
        indentStack.Push(currentIndent);
        yield return new Token(TokenType.INDENT);
    } else if (currentIndent < lastIndent) {
        while (currentIndent < indentStack.Peek()) {
            indentStack.Pop();
            yield return new Token(TokenType.DEDENT);
        }
    }
}

After each statement, optionally expect INDENT/DEDENT and use them to form block structures.