Deflake round 2 nodewise recovery#30779
Open
joe-redpanda wants to merge 14 commits into
Open
Conversation
Contributor
Author
|
/ci-repeat 5 dt-repeat=20 tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_recovery_local_data_missing |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the cloud topics L1 domain supervisor’s internal-topic reconciliation so it responds immediately to raft0 leadership and cluster membership changes, avoiding long delays (10min polling interval) where internal cloud-topics metastore topics can remain at an undesired replication factor after bootstrap/scale-out.
Changes:
- Trigger internal topic reconciliation on controller (raft0) leadership changes and members-table updates, rather than relying solely on periodic polling.
- Run reconciliation only on the raft0 leader to avoid redundant reconfiguration attempts from multiple nodes.
- Serialize and safely cancel/restart reconciliation work using
ssx::single_fiber_executorto prevent concurrent reconciliation runs.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/v/cloud_topics/level_one/domain/domain_supervisor.cc |
Adds event-driven reconciliation triggers, leader-gating, and a single-fiber executor to cancel/resubmit reconciliation safely. |
src/v/cloud_topics/level_one/domain/BUILD |
Adds build deps needed for partition leader notifications and single_fiber_executor. |
andrwng
reviewed
Jun 12, 2026
Collaborator
Retry command for Build#85684please wait until all jobs are finished before running the slash command |
Collaborator
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
from
June 12, 2026 21:43
151fa2f to
1ec6529
Compare
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
3 times, most recently
from
June 12, 2026 22:27
e6e6c3b to
4a28210
Compare
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
commented
Jun 12, 2026
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
from
June 12, 2026 22:53
4a28210 to
917d435
Compare
Contributor
Author
|
/ci-repeat 5 dt-repeat=20 tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_recovery_local_data_missing |
dotnwat
reviewed
Jun 16, 2026
dotnwat
left a comment
dotnwat
left a comment
Member
There was a problem hiding this comment.
i can't quite tell if this pr needs to be split up into separate commits, or if the hunks need to be moved around a bit so that the diff doesn't have so many interleaved changes. otherwise, a live walkthrough might be helpful.
andrwng
reviewed
Jun 17, 2026
andrwng
reviewed
Jun 17, 2026
andrwng
reviewed
Jun 17, 2026
use simple_time_jitter for the sleep between reconcile passes instead of a fixed interval.
subscribe to the node-local abort source so shutdown cancels the executor as soon as it begins rather than waiting for stop() to drain, and refuse to submit new runs once an abort has been requested.
the reconcile loop already sleeps once between passes; a 10s sleep after every replication factor bump only delayed the remaining topics in the pass. one sleep per pass, not one per topic.
thread the executor's abort source through do_reconcile into ensure_topic_replication so draining the executor (or a later resubmission) stops an in-flight pass at the next topic instead of letting it run to completion.
replace the conservative one-topic-at-a-time chain with a parallel pass over all internal topics, waited on in bulk.
adds notifications on raft0 leadership & node add / remove changes which interrupt any in-flight run and launch another reconciliation round through the single fiber executor. the reconcile loop now exits on non-leaders instead of sleeping and polling again: a later leadership change notification resubmits it.
add metastore reconciliation to the topics managed by health manager
The health_manager now reconciles the cloud-topics metastore topic's replication factor
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
from
July 21, 2026 21:54
fb02242 to
63e5809
Compare
joe-redpanda
commented
Jul 21, 2026
Collaborator
Retry command for Build#87503please wait until all jobs are finished before running the slash command |
joe-redpanda
commented
Jul 22, 2026
| backoff_sec: float = 2, | ||
| logger: Logger | None = None, | ||
| ) -> None: | ||
| """Wait until every health-manager-managed internal topic that currently |
Contributor
Author
There was a problem hiding this comment.
this is way too much prose make this terse
joe-redpanda
commented
Jul 22, 2026
| are created lazily are skipped until they exist. | ||
| """ | ||
|
|
||
| def all_at_target() -> bool: |
Contributor
Author
There was a problem hiding this comment.
this probably belongs in the admin client
joe-redpanda
commented
Jul 22, 2026
|
|
||
| def wait_for_internal_topic_replication( | ||
| admin: "Admin", | ||
| replication_factor: int = 3, |
Contributor
Author
There was a problem hiding this comment.
can we scrape this from redpanda's settings
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
from
July 22, 2026 18:47
4547472 to
01e01f1
Compare
joe-redpanda
commented
Jul 22, 2026
| ] | ||
|
|
||
|
|
||
| def wait_for_internal_topic_replication( |
Contributor
Author
There was a problem hiding this comment.
again this should go in the admin client
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
from
July 22, 2026 18:54
01e01f1 to
214eb70
Compare
joe-redpanda
commented
Jul 22, 2026
| ) -> None: | ||
| """Wait for every existing health-manager-managed internal topic to | ||
| reach the target replication factor. Skips topics not yet created and | ||
| ignores in-flight reconfigurations, so it is safe to call under load. An |
Contributor
Author
There was a problem hiding this comment.
why are you talking about what this DOESN't do
joe-redpanda
commented
Jul 22, 2026
| # cloud-topics metastore, born rf=1) have reached their target | ||
| # replication factor before we start decommissioning nodes. An rf=1 | ||
| # internal partition cannot be evacuated from a draining node, so a | ||
| # decommission that races the health manager's rf reconciliation stalls |
Contributor
Author
There was a problem hiding this comment.
This comment should be a one line on 'wait for internal topics to bootstrap to full replication factor or decommission will stall'
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
2 times, most recently
from
July 22, 2026 19:11
a5b507f to
1b290ce
Compare
joe-redpanda
commented
Jul 22, 2026
| , _as(as) {} | ||
|
|
||
| ss::future<> health_manager::start() { | ||
| _as_sub = _as.local().subscribe( |
Contributor
Author
There was a problem hiding this comment.
the abort source should be getting cancelled on stop
nothing has consumed the boolean since the sequential reconcile chain was replaced with a parallel pass; each topic's outcome is already logged.
Add three Admin methods for reasoning about internal-topic replication: - internal_topic_replication_factor(): reads the internal_topic_replication_factor cluster config. - all_partitions_have_replicas(topic, rf): whether every partition of a topic has at least rf replicas; a missing topic is vacuously satisfied. - wait_for_internal_topic_replication(): wait for every existing health-manager-managed internal topic (listed in HEALTH_MANAGER_INTERNAL_TOPICS, mirroring the internal_topics array in cluster/health_manager.cc) to become fully replicated at the target rf. get_partitions reports the target replica set as soon as a move is issued, so this also requires the topic to have no in-flight reconfiguration; reconfigurations of other topics are ignored, so it is safe to call under load. An rf=1 internal partition cannot be evacuated off a draining node, so callers use this before decommissioning.
ShadowLinkingRandomOpsTest.test_node_operations started the workload and node-operation loop with no barrier, so a decommission could target the node holding the sole rf=1 replica of the cloud-topics metastore (ct_l1_domain) before the health manager reconciled it up. That partition cannot be evacuated, so the decommission stalls and the test fails with "decommissioning stopped making progress". Wait for the health-manager-managed internal topics to reach their target replication factor before the node-ops loop. This only waits on internal-topic rf, not for the cluster to quiesce, so the workload's in-flight reconfigurations keep running -- the flux is the point of this test.
joe-redpanda
force-pushed
the
deflake_round_2_nodewise_recovery
branch
from
July 22, 2026 19:53
1b290ce to
2642add
Compare
Contributor
Author
|
@dotnwat split the changes to health manager. swatted some additional bugs in the same vein. I'm probably going to leave this until after we cut the new release branch so these can bake, changing health monitor has kicked the rug and sent all the bugs scurrying. |
Collaborator
Retry command for Build#87583please wait until all jobs are finished before running the slash command |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes my prior attempted deflake of nodewise recovery.
The issue is as follows:
Clusters bootstrap with 1 node initially, and then grow to their target size by joining new nodes.
Cloud topic internal topics start eagerly rather than lazily like other topics.
The cloud topics internal topics may bootstrap while the cluster is still single node, at which point rather than honoring internal_replication, they fall back to rf=1.
The issue, though, is that reconciliation of the cloud topics internal topics happens only once every (hardcoded) 10 minutes, and does NOT shortcut on membership changes.
As a result, if the cloud topics internal topic bootstraps with rf=1, it will be stuck like that for 10 minutes.
My prior change modified the nodewise recover test to wait for the cluster to be 'settled' aka for there to be no reconfigurations and for the cloud topics internal topics to have reached rf=3: this settling evidently could take up to 10 minutes, which way overwhelmed the timeout of 2 minutes I had assigned.
This PR rectifies the above by having internal reconciliation shortcut when membership changes and moving cloud topics internal topic over to the health manager which reconciles the others
This involved a few other changes and cleanups:
Addendum:
Adds in a similar fix to what was in nodewise recovery tests to shadow linking chaos tests.
Adds the admin function to wait for internal topic replication before launching into a bunch of node decommissions.
Backports Required
Release Notes
Improvements