Skip to content

RebuildInstance ops with nonexistent backupName retries forever in the in-place path #10542

Description

@weicao

Problem

A RebuildInstance OpsRequest using the in-place path (rebuildFrom[].inPlace: true) with a backupName that references a nonexistent Backup never fails: the ops controller retries the reconcile forever and the OpsRequest stays in Running.

Scenario

  • User submits an in-place rebuild from a backup, but the backup name is typo'd, or
  • the referenced Backup object was deleted after the OpsRequest was submitted.

Either way the Backup does not exist when the reconcile path resolves it — and it will never appear, so retrying cannot help.

Deterministic reproducibility

Webhook/validation does not guard this: validateRebuildInstance (pkg/operations/validation.go) only validates the component/instances, not the referenced backup. So a missing backup at reconcile time is a deterministic, permanent condition, not a transient one.

Reproduce

  1. Create a cluster with a component that supports rebuild.
  2. Create a RebuildInstance OpsRequest with inPlace: true and backupName: does-not-exist.
  3. Observe the OpsRequest stays Running forever; the controller keeps requeueing on a backups.dataprotection.kubeblocks.io "does-not-exist" not found error.

Framework mechanics / root cause

In pkg/operations/rebuild_instance.go, prepareInplaceRebuildHelper:

  • the cli.Get of the Backup named by rebuildFrom[].backupName returns the bare NotFound error (~L598);
  • the ActionSet named by backup.Status.BackupMethod.actionSetName is fetched via dputils.GetActionSetByName, which also returns the bare NotFound error (~L610).

In the operations framework, a plain error from ReconcileAction means "transient, requeue and retry"; only intctrlutil.NewFatalError marks the instance progress as Failed (see rebuildInstancesInPlace, which catches ErrorTypeFatal and sets FailedProgressStatus, but returns any other error for requeue). The adjacent validations in the same function (backup not Full/Incremental, backup not Completed, empty backupMethod) already return NewFatalError; only these two NotFound lookups leak through as retryable.

Relationship to #10510

#10510 fixed a different lookup in a different phase: the instance (pod) lookup in the Action phase. This issue is about the backup / actionset lookup in the reconcile phase (ReconcileActionrebuildInstancesInPlaceprepareInplaceRebuildHelper), which #10510 did not touch.

Expected behavior

A nonexistent backup (or a deleted ActionSet referenced by the backup) should be classified as a fatal error so the affected instances are marked Failed and the OpsRequest transitions to Failed, instead of retrying forever.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions