CASSANDRA-21134: Direct I/O for background SSTable writes by samueldlightfoot · Pull Request #4815 · apache/cassandra

samueldlightfoot · 2026-05-17T09:18:21Z

CASSANDRA-21134: Direct I/O for background SSTable writes

Summary

Opt-in O_DIRECT write path for background SSTable producers, bypassing the OS page cache for write-once read-never data. Memtable flushes remain buffered (hot data benefits from the cache).

background_write_disk_access_mode: direct    # default: standard
direct_write_buffer_size: 1MiB                # aligned up to FS block size; auto-grows to chunk_length

Gated by (1) config, (2) table compression enabled, (3) an OperationType allowlist (DataComponent#DIRECT_WRITE_SUPPORT). Selection is central in DataComponent.buildWriter; producers are unchanged.

Performance

Benchmark results are attached to the JIRA. Significant p99 read latency improvements under throttled compaction.

Operations covered (DIO eligible)

OperationType	End-to-end test
`WRITE`	`CQLSSTableWriterDaemonTest` (parameterised on disk mode)
`COMPACTION`	`CompactionsTest` (parameterised on disk mode)
`MAJOR_COMPACTION`	`CompactionsTest.testCompactionWithSizeLimitedRewriter`
`CLEANUP`, `GARBAGE_COLLECT`, `TOMBSTONE_COMPACTION`, `UPGRADE_SSTABLES`	`CompactionsTest` (transitive)
`ANTICOMPACTION`	`AntiCompactionTest.testAntiCompactionWithCompressedTableAndDirectWrites`
`STREAM`	`StreamingDirectWriteTest`

The allowlist is exhaustive: any new OperationType with writesData == true that is not classified fails static initialization (AssertionError).

Operations NOT covered

Path	Classification	Reason	Coverage
`FLUSH` (memtable)	`UNSUPPORTED_POLICY`	Just-flushed data is hot — keep it in the page cache.	`DataComponentDirectWriteSelectionTest`
`SCRUB`	`UNSUPPORTED_CORRECTNESS`	`tryAppend` needs `mark()` / `resetAndTruncate()`, which DIO cannot satisfy.	`DataComponentDirectWriteSelectionTest`
Zero-Copy Streaming	n/a (path bypass)	Entire-SSTable streaming bypasses `DataComponent.buildWriter`.	`StreamingDirectWriteTest` (disables ZCS)
Uncompressed writers	n/a (path bypass)	Only `CompressedSequentialWriter` has a DIO subclass.	`DataComponentDirectWriteSelectionTest` (compression gate)

Removing an UNSUPPORTED_CORRECTNESS entry requires code changes; UNSUPPORTED_POLICY is a policy decision.

Key code

io/DirectIoSupport.java — eligibility enum (SUPPORTED / UNSUPPORTED_CORRECTNESS /
UNSUPPORTED_POLICY / NOT_APPLICABLE).
io/sstable/format/DataComponent.java — selection, allowlist, exhaustiveness check;
per-op first-activation log.
io/compress/DirectCompressedSequentialWriter.java — new writer; aligned buffers,
mark()/resetAndTruncate() unsupported.
io/compress/CompressedSequentialWriter.java — refactored so the DIO subclass can
override the write-chunk path; writeChunk contract documented and asserted.
config/Config.java, config/DatabaseDescriptor.java — new knobs, validation, startup
wiring; buffer size aligned to FS block size, auto-grown to chunk length.
service/StartupChecks.java — fails fast if direct is requested on a platform/FS
that does not support O_DIRECT.

Tests introduced

Property-based DIO-writer sweep — write/read integrity and on-disk byte-identity
vs. the buffered writer over compressors × chunk lengths × random payload sizes;
seed-logged for repro (DirectCompressedSequentialWriterTest).
Parameterised buffer-size tests — three regimes pinning the distinct branches of
flushCompleteBlocks (DirectCompressedSequentialWriterTest).
Selection-matrix tests — per-OperationType eligibility, allowlist exhaustiveness,
compression gate, config-mode gate (DataComponentDirectWriteSelectionTest).
End-to-end coverage per allowlist arm — WRITE (CQLSSTableWriterDirectWriteTest),
STREAM (in-JVM dtest, StreamingDirectWriteTest), and the compaction family +
ANTICOMPACTION (extended CompactionsTest, AntiCompactionTest).
Regression guards — constructor channel-leak protection, non-power-of-two
block-size rejection, once-per-JVM undersized-buffer warn, SCRUB-gating canaries
(DirectCompressedSequentialWriterTest).
Resource-leak detection — BufferPoolMXBean check that the off-heap aligned
buffer is returned on close (DirectCompressedSequentialWriterTest).
Config validation — new YAML knobs (mode parsing, buffer-size bounds, defaults)
in DatabaseDescriptorTest.

Not in scope

Uncompressed SSTable writers.
ZCS streaming.

Reviewer notes

Findings from the Cassandra bug-hunting skills (Opus 4.7 xhigh & kimi-k2.6:cloud) were addressed prior to
review.

Adds an opt-in O_DIRECT write path for background SSTable producers, bypassing the OS page cache for data that is unlikely to be re-read soon after being written. Memtable flushes remain buffered. Enabled via two new YAML knobs: - background_write_disk_access_mode: standard (default) | direct - direct_write_buffer_size: 1MiB (default; aligned up to FS block size, auto-grown to chunk_length) The path is gated by config, table compression being enabled, and an OperationType allowlist in DataComponent. The allowlist is exhaustive: any new OperationType with writesData=true that is not classified will fail static initialization. Operations on the DIO path: COMPACTION, MAJOR_COMPACTION, TOMBSTONE_COMPACTION, ANTICOMPACTION, GARBAGE_COLLECT, CLEANUP, UPGRADE_SSTABLES, WRITE, STREAM (chunked receiver only). Operations off the DIO path: - FLUSH (policy: just-flushed data is hot, keep in page cache) - SCRUB (correctness: tryAppend needs mark/resetAndTruncate) - Zero-Copy Streaming (bypasses DataComponent.buildWriter) - Uncompressed writers (only CompressedSequentialWriter has a DIO subclass in this change) StartupChecks fails fast if 'direct' is requested on a platform/FS that does not support O_DIRECT. patch by Sam Lightfoot; reviewed by <reviewers> for CASSANDRA-21134

Fail fast on unsupported background_write_disk_access_mode values, drop "scrub" from the DIO operation docs (it's UNSUPPORTED_CORRECTNESS), and tighten AntiCompactionTest/CompactionsTest to assert Direct I/O actually engages rather than passing spuriously.

…st.yaml Add background_write_disk_access_mode: direct to cassandra_latest.yaml so new-install/latest configurations exercise Direct I/O by default, and keep the test latest config in sync (test/conf/latest_diff.yaml and the DTEST_JVM_DTESTS_USE_LATEST block in InstanceConfig.java). The cassandra.yaml default remains standard.

aweisberg

Mostly just nits but at least one or two things worth doing and then I shared a pastebin with you of an analysis of the DirectCompressedSequentialWriterTest code and boundary coverage that is worth considering.

aweisberg · 2026-05-28T21:42:07Z

+# (compaction, streaming, scrub, cleanup, repair, etc.). The allowed values are:
+# - standard: use buffered I/O (default)
+# - direct: use direct I/O, bypassing the OS page cache
+# Note: Only applies to compressed tables. Uncompressed tables always use buffered I/O.


You don't really need to prefix with Note:. It's not really information bearing.

aweisberg · 2026-05-28T21:45:36Z

+
+        if (providedMode == DiskAccessMode.auto)
+        {
+            providedMode = DiskAccessMode.standard;


Auto seems pretty unconditional here so the check above to log that it changed doesn't seem to mean much? I guess it clarifies that AUTO always resolves to STANDARD. Not sure when it would change in the future.

It's fine just something I noted.

aweisberg · 2026-05-28T22:12:12Z


        this.sstableMetadataCollector = sstableMetadataCollector;
-        crcMetadata = new ChecksumWriter(new DataOutputStream(Channels.newOutputStream(channel)));
+        crcMetadata = new ChecksumWriter(new DataOutputStream(Channels.newOutputStream(this.channel)));


Why add this? I think our code style even mentions avoiding this when it isn't needed.

aweisberg · 2026-05-29T16:34:39Z

+                && isDirectWriteSupported(operationType))
+            {
+                if (directWriteLogged.add(operationType))
+                    logger.info("Direct I/O writer activated for {}", operationType);


This is a bit quirky for an info level log? It means it happened once, but doesn't tell you if other attempts in different paths would also have succeeded.

aweisberg · 2026-05-29T16:56:10Z

+    // fit contiguously. flushCompleteBlocks aligns down to blockSize, leaving up to
+    // (blockSize - 1) bytes carried over via compact(); the floor accounts for that.
+    private ByteBuffer writeBuffer;
+    private int writeBufferPosition = 0;


Trying to figure out why this is tracked separately from ByteBuffer.position()

aweisberg · 2026-05-29T17:46:04Z

    }
+
+    @Test
+    public void testInitializeBackgroundWriteDiskAccessModeRejectsZeroBufferSize()


Technically haven't exercised the checks of the paths of isDirectIOSupported which would require being able to intercept that and provide test answers to whether paths support that. Maybe something that can be done with jimfs.

The auto path which we discussed might be worth changing.

aweisberg · 2026-05-29T17:48:20Z

+                        + ") WITH compression = {'enabled':'true', 'class':'LZ4Compressor'}";
+        String insert = "INSERT INTO " + qualifiedTable + " (k, v) VALUES (?, ?)";
+
+        DirectIoTestUtils.activatedDirectWriteOperations().clear();


Oof. I get the pain of trying to observe this side effect. This is something jimfs might be able to help with, a mock file system would let you spy on what it is doing.

aweisberg · 2026-05-29T17:53:39Z

+
+        if (DatabaseDescriptor.getBackgroundWriteDiskAccessMode() == DiskAccessMode.direct)
+        {
+            Set<OperationType> activated = DirectIoTestUtils.activatedDirectWriteOperations();


Should assert the negative as well when non-direct is requested?

aweisberg · 2026-05-29T18:01:41Z

+
+import static org.junit.Assert.assertTrue;
+
+public class DataComponentDirectWriteSelectionTest


The negative checks are enumerated instead of being derived from including everything that is not in SUPPORTED_OPS so it's not going to exercise everything as new components are added.

It also doesn't fail if an unrecognized component should succeed.

aweisberg · 2026-05-29T18:05:01Z

+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class StreamingDirectWriteTest extends TestBaseImpl


Curious why Streaming needed a dedicated test. Incoming buffers is just incoming buffers right? Not complaining about the coverage though.

samueldlightfoot force-pushed the CASSANDRA-21134-direct-compaction-writes branch from 3349322 to 005f4e1 Compare May 17, 2026 09:25

samueldlightfoot marked this pull request as ready for review May 17, 2026 09:25

samueldlightfoot force-pushed the CASSANDRA-21134-direct-compaction-writes branch 4 times, most recently from b95f990 to 3d95be1 Compare May 18, 2026 09:58

samueldlightfoot force-pushed the CASSANDRA-21134-direct-compaction-writes branch from a49806c to ca8ef09 Compare May 19, 2026 17:23

aweisberg self-requested a review May 28, 2026 20:46

samueldlightfoot added 2 commits May 29, 2026 08:17

aweisberg requested changes May 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CASSANDRA-21134: Direct I/O for background SSTable writes#4815

CASSANDRA-21134: Direct I/O for background SSTable writes#4815
samueldlightfoot wants to merge 3 commits into
apache:cassandra-6.0from
samueldlightfoot:CASSANDRA-21134-direct-compaction-writes

samueldlightfoot commented May 17, 2026 •

edited

Loading

Uh oh!

aweisberg left a comment

Uh oh!

aweisberg May 28, 2026

Uh oh!

aweisberg May 28, 2026

Uh oh!

aweisberg May 28, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

aweisberg May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		import static org.junit.Assert.assertTrue;

		public class DataComponentDirectWriteSelectionTest


		import static org.assertj.core.api.Assertions.assertThat;

		public class StreamingDirectWriteTest extends TestBaseImpl

Conversation

samueldlightfoot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CASSANDRA-21134: Direct I/O for background SSTable writes

Summary

Performance

Operations covered (DIO eligible)

Operations NOT covered

Key code

Tests introduced

Not in scope

Reviewer notes

Uh oh!

aweisberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

samueldlightfoot commented May 17, 2026 •

edited

Loading