Skip to content

fix: clean up deleting restore PVCs - #10605

Closed
weicao wants to merge 3 commits into
mainfrom
bugfix/task139-pvc-delete-finalizer
Closed

fix: clean up deleting restore PVCs#10605
weicao wants to merge 3 commits into
mainfrom
bugfix/task139-pvc-delete-finalizer

Conversation

@weicao

@weicao weicao commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • handle a deleting restore PVC before validating its source Backup
  • stop and wait for the execution Restore before removing populate dependents
  • keep the target PVC finalizer while owned populate Jobs or the helper PVC still exist
  • release the target finalizer only after a fresh reconcile confirms the dependents are absent
  • propagate every transient deletion-cleanup API failure through the outer reconciler even when the PVC already has a Populating condition
  • preserve the existing non-deletion cleanup behavior for bound PVC completion

Fixes #10606.

Retry contract

A deleting PVC is still owned by the volume-populator controller until cleanup converges. Ordinary API errors from execution-Restore cleanup, Job cleanup, helper-PVC cleanup, or target-finalizer removal must therefore reach controller-runtime and be retried. The outer handler keys this rule from deletionTimestamp; it deliberately does not re-check the in-memory finalizer because Cleanup removes that local value before attempting the final Patch, and a failed Patch means the persisted finalizer still exists.

The existing Populating-condition suppression remains limited to non-deleting PVCs whose external controller owns subsequent progress.

Tests

  • focused cleanup tests cover deletion ordering, dependent absence, and all eight API failure points
  • the eight-point failure matrix enters through the complete Reconcile path, asserts the injected error reaches controller-runtime, and proves the persisted target finalizer remains
  • KUBEBUILDER_ASSETS='<envtest 1.26.1 assets>' go test ./controllers/dataprotection -count=1
  • focused go test -race for deletion and outer-handler cases
  • go vet ./controllers/dataprotection
  • git diff --check

Validation on exact head 35f7f55c14e6c76208aab161d11c3ad35313e359 is green. Runtime environment coverage remains N=0.

Supersedes #10604, which was closed when its non-conforming source branch was renamed.

@apecloud-bot

Copy link
Copy Markdown
Collaborator

Auto Cherry-pick Instructions

Usage:
  - /nopick: Not auto cherry-pick when PR merged.
  - /pick: release-x.x [release-x.x]: Auto cherry-pick to the specified branch when PR merged.

Example:
  - /nopick
  - /pick release-1.1

CLA Recheck Instructions

Usage:
  - /recheck-cla: Trigger a re-check of CLA status for this pull request.
Example:
  - /recheck-cla

@github-actions github-actions Bot added the size/L Denotes a PR that changes 100-499 lines. label Jul 10, 2026
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.77419% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.94%. Comparing base (12611aa) to head (35f7f55).
⚠️ Report is 34 commits behind head on main.

Files with missing lines Patch % Lines
...llers/dataprotection/volumepopulator_controller.go 96.77% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10605      +/-   ##
==========================================
- Coverage   63.97%   63.94%   -0.03%     
==========================================
  Files         520      520              
  Lines       62887    62942      +55     
==========================================
+ Hits        40229    40249      +20     
- Misses      19071    19101      +30     
- Partials     3587     3592       +5     
Flag Coverage Δ
unittests 63.94% <96.77%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weicao weicao added the nopick Not auto cherry-pick when PR merged label Jul 16, 2026
Handle deletion before source validation so a missing backup cannot strand the data protection finalizer. Keep the target finalizer until owned populate jobs and the helper PVC are confirmed absent on a fresh reconcile.
@weicao
weicao force-pushed the bugfix/task139-pvc-delete-finalizer branch from 0e627e7 to abef483 Compare July 16, 2026 18:09
@weicao
weicao marked this pull request as ready for review July 16, 2026 18:09
@weicao
weicao requested review from a team, ldming and wangyelei as code owners July 16, 2026 18:09
Comment thread controllers/dataprotection/volumepopulator_controller.go
Comment thread controllers/dataprotection/volumepopulator_controller.go
Return transient cleanup failures through the outer reconcile handler even when the deleting PVC already has a Populating condition. Exercise every cleanup API failure through Reconcile so the finalizer cannot become stranded.

Refs #10606
@leon-ape

Copy link
Copy Markdown
Collaborator

Closing this PR as superseded by #10777.

#10605 established several useful cleanup invariants: stop the execution Restore before dependent cleanup, observe Restore/helper absence before releasing the target PVC finalizer, and keep cleanup API failures retryable. Those invariants should be retained, but the cleanup must be coordinated by the Cluster restore lifecycle rather than by a parallel PVC-local deletion workflow.

#10777 provides the stronger ownership model: a Cluster restore finalizer, Cluster UID identity, verified owner references, direct API-server reads, ordered Cluster-wide teardown, and legacy-resource adoption. It is now the authoritative implementation path. Direct deletion of an active restore target PVC is treated as an unsupported/destructive management operation and should remain fail-closed instead of triggering autonomous DP cleanup.

Any remaining safety checks and failure-path tests from this PR should be carried into #10777 where they apply. No commits from this PR need to be merged directly.

@leon-ape leon-ape closed this Aug 28, 2026
@github-actions github-actions Bot added this to the Release 1.2.0 milestone Aug 28, 2026
@leon-ape
leon-ape deleted the bugfix/task139-pvc-delete-finalizer branch August 28, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nopick Not auto cherry-pick when PR merged size/L Denotes a PR that changes 100-499 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Volume populator strands deleting PVC when source Backup is missing

3 participants