Fix foreach macro range preprocessing

[azwolski]

Description

Fixes #3809.

Clang tokenizes cases like 0...MAXI as a single preprocessing-number token, so MAXI is not expanded as a macro. This breaks foreach ranges when there is no space between the numeric start value, the ... operator, and the macro upper bound.

This patch normalizes the main input file before running Clang's preprocessor. It uses Clang's raw lexer to find numeric tokens containing ... followed by an identifier-start character, then inserts a space before the ellipsis:

0...MAXI -> 0 ...MAXI

Related Issue

• Linked to relevant issue(s)

Checklist

• Code has been formatted with clang-format (e.g., clang-format -i src/ispc.cpp)
• Git history has been squashed to meaningful commits (one commit per logical change)
• Compiler changes are covered by lit tests
• Language/stdlib changes include new functional tests for runtime behavior
• Documentation updated if needed

[Copilot]

Pull request overview

This PR fixes a Clang preprocessor corner case that breaks foreach ranges like 0...MAXI when MAXI is a macro, by normalizing the main input file prior to preprocessing so that ... is tokenized separately and the macro expands correctly.

Changes:

• Added a raw-lexer based normalization pass that inserts a space before ... when it appears inside a numeric_constant token followed by an identifier-start character.
• Updated preprocessor setup to feed Clang a normalized main file buffer (including stdin handling).
• Added lit coverage for the regression (and an explicit XFAIL documenting the current included-file limitation).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/preprocessor.cpp	Adds source normalization using Clang’s raw lexer and applies it to the main input before preprocessing.
tests/lit-tests/3809.ispc	Regression test ensuring foreach(i = 0...MACRO) compiles and comments are not rewritten.
tests/lit-tests/3809-include-limitation.isph	Header used to demonstrate the current limitation for included files.
tests/lit-tests/3809-include-limitation.ispc	XFAIL lit test documenting that included files aren’t normalized yet.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zephyr111]

Looks good to me (besides the TODO note)!
I did not found any issues while performing some additional tests.

[zephyr111]

Do you plan to add this in another PR or later in this one?

[azwolski]

I would like to try

[azwolski]

Would like me to remove the TOODs?

[aneshlya]

Please remove TODO and create an issues instead. Thanks for creating lit test for this.

[dbabokin]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aneshlya]

Please remove TODO and create an issues instead. Thanks for creating lit test for this.

[elena901]

A little bit late with it, but i like this option.
Richard Jerome:
Fix the pre-processor in the clang code itself so for it not to recognize this as a single token.

[aneshlya]

A little bit late with it, but i like this #3809 (comment).

@elena901, this PR already implements the option 1 and it's not relevant place for such comment, it should go to the linked issue.