Stream encoding conversion in CSV reader to reduce peak memory

[kemo]

Summary

• Replaces file_get_contents() + full-string encoding conversion with chunk-based streaming (64KB chunks)
• Handles multi-byte character boundaries for UTF-16 and UTF-32 encodings by carrying leftover bytes between chunks
• Uses php://temp stream which spills to disk if memory threshold is exceeded
• Reduces peak memory from 2x file size to ~128KB constant overhead

Changes

• src/PhpSpreadsheet/Reader/Csv.php: New convertEncodingStreaming() method with encodingCharWidth() helper, replaces the old file_get_contents path in openFileOrMemory()
• tests/PhpSpreadsheetTests/Reader/Csv/CsvStreamingEncodingTest.php: 18 tests covering ISO-8859-1, UTF-16BE/LE, UTF-32BE/LE, Windows-1252, large files (>64KB), chunk boundary handling
• tests/PhpSpreadsheetTests/Benchmark/CsvStreamingEncodingBenchmark.php: Memory scaling benchmark for large non-UTF-8 CSVs

Test plan

• Verify non-UTF-8 CSV files (ISO-8859-1, UTF-16, UTF-32, Windows-1252) read correctly
• Verify large files with encoding conversion produce identical results
• Verify UTF-8 files are unaffected (no conversion path taken)
• Verify multi-byte characters spanning chunk boundaries are handled correctly
• Run benchmark: vendor/bin/phpunit --group benchmark --filter CsvStreamingEncodingBenchmark --stderr

[kemo]

Benchmark Results (csv-streaming-encoding)

Targeted benchmark: read ISO-8859-1 and UTF-16LE encoded CSV files at scale (20 columns).

Benchmark	Master	Branch	Delta
5k rows ISO-8859-1 (4.1MB)	4,323ms	4,807ms	+11%
25k rows ISO-8859-1 (21MB)	21,340ms	21,768ms	+2%
50k rows	OOM	OOM	Both OOM at 512MB

No measurable time or memory improvement at these sizes. Both branches OOM at 50k rows — the bottleneck is the Spreadsheet object, not the encoding conversion. The streaming approach may show benefits on files where the old code materialised the entire encoded content in memory before parsing.

[oleibman]

Thank you for the summaries. For purposes of prioritization, I am going to put the PRs with less demonstrable improvement, including this one, in draft status for now. This does not mean that I will not return to them.

[oleibman]

I believe your chunking will fail if the encoding is a UTF-16 variant and the chunk winds up splitting a surrogate pair. And, not that we want to encourage it, it will often fail for UTF-7 as well.

[kemo]

Good catch — the chunked approach doesn't account for multi-byte character boundaries. UTF-16 surrogate pairs and UTF-7 shift sequences would both break if a chunk splits mid-character. I'll address this if the PR is revisited.

[oleibman]

As we discussed this, it seemed to me that there is a case for making Csv Reader's encoding conversion an overridable function. This is done in PR #4845. If that change is merged, this one will probably end up with an easily resolved merge conflict. As part of the tests for that change, I introduce a new test class which shows how you might be able to leverage a Php Iconv Filter to reduce both memory consumption and execution time. See tests/PhpSpreadsheetTests/CsvIconv2 and NonUtf8Test. I'm not sure I want to travel down that road as a permanently supported feature (I don't know much about filters and don't want to have to spend time supporting them), but it should lay the groundwork for making it easier for you to test and make suggestions.

[oleibman]

As predicted, there were merge conflicts, which I have now resolved. This does not mean that I am more inclined to move forward with this change than I was, but you should be able to experiment further using the current state of the PR as a starting point.