Restore broken on 1.2.0-alpha.3: restore-from-backup annotation silently ignored, and spec.restore populator PVC is deleted underneath its own prepareData job

[dudizimber]

Restore is broken on 1.2.0-alpha.3: the annotation path is silently ignored and the spec.restore populator PVC is deleted underneath its own prepareData job

Summary

On 1.2.0-alpha.3 there is no working way to restore a Cluster from a Backup, and both failure modes are silent — the Cluster reaches status.phase: Running with an empty database, so nothing surfaces the problem to the user.

1. kubeblocks.io/restore-from-backup is ignored entirely — no Restore object is created at all. This worked on 1.2.0-alpha.1.
2. Cluster.spec.restore does create a Restore, but its prepareData job can never run because the kb-populate-<uid> PVC it is supposed to mount is deleted (or never created). The Restore stays Running / PrepareData=False indefinitely.

In both cases the database pods start and begin serving an empty dataset while the restore is still incomplete. A user watching kubectl get cluster sees Running and would reasonably conclude the restore succeeded.

This is distinct from #10749 (restore-manager sidecar never receiving its stop signal), which is closed. The sidecar no longer hangs; the failure has moved to the populator PVC lifecycle.

Environment

KubeBlocks	1.2.0-alpha.3 (helm chart kubeblocks-1.2.0-alpha.3)
Kubernetes	v1.31.5+k3s1 (k3d, single node)
Addon	falkordb chart 1.6.5, appVersion 4.20.1
StorageClass	local-path (default)
BackupRepo	MinIO-backed, Ready

The Backup itself is healthy and is not implicated:

NAME                  METHOD     REPO                STATUS      TOTAL-SIZE   DURATION
fdb-backup-datafile   datafile   falkordb-e2e-repo   Completed   2724         12s

Case 1 — kubeblocks.io/restore-from-backup is silently ignored

Create a Cluster carrying the annotation:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: fdb-restored
  annotations:
    kubeblocks.io/restore-from-backup: '{"falkordb":{"name":"fdb-backup-datafile","namespace":"<ns>","volumeRestorePolicy":"Parallel"}}'
spec:
  terminationPolicy: WipeOut
  clusterDef: falkordb
  topology: replication
  componentSpecs:
    - name: falkordb
      replicas: 2
      # ... volumeClaimTemplates: data, 1Gi
    - name: falkordb-sent
      replicas: 3

The annotation is present on the created object:

$ kubectl get cluster fdb-restored -n <ns> -o jsonpath='{.metadata.annotations}'
{"kubeblocks.io/restore-from-backup":"{\"falkordb\":{\"name\":\"fdb-backup-datafile\",...}}", ...}

But no restore is ever attempted:

$ kubectl get restores.dataprotection.kubeblocks.io -n <ns>
No resources found in <ns> namespace.

$ kubectl get jobs -n <ns>
No resources found in <ns> namespace.

The Cluster nonetheless goes Running with a correct-looking topology and no data:

$ kubectl get cluster fdb-restored -n <ns>
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
fdb-restored   falkordb             WipeOut              Running   88s

# 1 primary + 1 secondary, replication link up, DBSIZE 0

Expected: either the annotation is honoured, or — if it has been intentionally removed in favour of spec.restore — the Cluster is rejected or a warning event is emitted. Silently dropping a restore request is the problem.

This is a regression: the same annotation on the same addon and the same Backup restores correctly on 1.2.0-alpha.1.

Case 2 — spec.restore hangs because the populator PVC is deleted

Same Backup, using the alpha.3 API instead:

spec:
  restore:
    source:
      apiGroup: dataprotection.kubeblocks.io
      kind: Backup
      name: fdb-backup-datafile
      namespace: <ns>

This is wired up correctly at first — Restore objects are created, and they target only the falkordb data PVC and leave the falkordb-sent PVCs alone, which is the right behaviour:

$ kubectl get restores.dataprotection.kubeblocks.io -n <ns>
NAME                                                      BACKUP                STATUS      DURATION
kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6          fdb-backup-datafile   Running
kb-populate-c821f723-5655-48e0-b6df-b28c39740e8c          fdb-backup-datafile   Running
restore-a9818ceb-c69c-48bc-9f67-7ecc000347f2-post-ready   fdb-backup-datafile   Completed   6s

The two kb-populate-* restores never leave Running:

phase: Running
  prepareData: status=Processing
    "Job/restore-preparedata-abd972dc-fdb-backup-datafile-0" is processing
  conditions:
    CheckBackupRepo    True
    ValidationPassed   True   validate restore spec successfully
    PrepareData        False  processing prepareData stage.

Their jobs can never be scheduled, because each one wants a kb-populate-* PVC that is gone:

$ kubectl describe pod restore-preparedata-abd972dc-fdb-backup-datafile-0-xbhdq -n <ns>
Warning  FailedMount  kubelet
  Unable to attach or mount volumes:
  unmounted volumes=[dp-claim-kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6]
  error processing PVC <ns>/kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6:
  PVC is being deleted

$ kubectl describe pod restore-preparedata-226c33a6-fdb-backup-datafile-0-5v6lf -n <ns>
Warning  FailedScheduling  default-scheduler
  running PreBind plugin "VolumeBinding": binding volumes:
  failed to check provisioning pvc: could not find v1.PersistentVolumeClaim
  "<ns>/kb-populate-c821f723-5655-48e0-b6df-b28c39740e8c"

So one job's populator PVC is terminating while the job tries to mount it, and the other job's populator PVC does not exist at all. The Restore has the PVC in its own spec.prepareDataConfig.volumeClaims[]:

"prepareDataConfig": {
  "volumeClaimRestorePolicy": "Parallel",
  "volumeClaims": [
    { "metadata": { "name": "kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6", "namespace": "<ns>" },
      "volumeClaimSpec": { "accessModes": ["ReadWriteOnce"], ... } }
  ]
}

Meanwhile the Cluster reports Running and the database serves an empty dataset:

$ kubectl wait --for=jsonpath='{.status.phase}'=Running cluster/fdb-r2 -n <ns> --timeout=240s
cluster.apps.kubeblocks.io/fdb-r2 condition met

$ kubectl exec fdb-r2-falkordb-0 -c falkordb -- redis-cli DBSIZE
0

Note the two independent problems here: the populator PVC race, and the fact that the workload is allowed to start and serve traffic while prepareData has not completed. Even once the PVC race is fixed, gating pod startup on restore completion would turn a silent empty database into an honest failure.

Expected behaviour

• spec.restore completes, prepareData finishes, and the Cluster comes up with the backed-up data.
• The Cluster does not reach Running while its restore is still in prepareData.
• kubeblocks.io/restore-from-backup either works or is explicitly rejected/warned about rather than silently dropped.

Impact

There is no working restore path on 1.2.0-alpha.3. Because both failures present as a healthy Running Cluster, a restore that silently produces an empty database is very easy to mistake for a successful one — this was only caught by an end-to-end test that asserts on the restored data rather than on status.phase.

[shanshanying]

HI @dudizimber
Thanks for your detailed report.

The legacy kubeblocks.io/restore-from-backup annotation was intentionally removed when restore was switched to Cluster.spec.restore. However, silently accepting the obsolete annotation is poor upgrade behavior.

@leon-ape please help check the spec.restore failure .

[dudizimber]

Thanks @shanshanying — that clears up case 1. One detail makes the upgrade behaviour worse than "the obsolete annotation is silently accepted", and it's worth having on the record before case 2 is triaged.

The annotation was removed in the same release that introduced its replacement, so there is no version where both work:

version	kubeblocks.io/restore-from-backup	Cluster.spec.restore
v1.1.0-beta.9	present	absent
v1.2.0-alpha.0	present	absent
v1.2.0-alpha.1	present	absent
v1.2.0-alpha.2	removed	present
v1.2.0-alpha.3	removed	present
main	removed	present

Checked against pkg/constant/annotations.go (RestoreFromBackupAnnotationKey = "kubeblocks.io/restore-from-backup", present at v1.2.0-alpha.1, gone at v1.2.0-alpha.2) and apis/apps/v1/cluster_types.go (spec.restore absent through v1.2.0-alpha.1, present from v1.2.0-alpha.2). I also confirmed the CRD side at runtime — on v1.2.0-alpha.1 neither v1 nor v1alpha1 has spec.restore in the Cluster schema, so it isn't merely unimplemented, it's unwritable.

Two consequences:

• No single Cluster manifest can restore on both sides of the boundary. An addon or operator supporting alpha.1 and alpha.2 has to branch on the KubeBlocks version to emit either the annotation or spec.restore. A release that kept honouring the annotation while spec.restore was the documented path would have avoided this.
• Because the annotation is dropped rather than rejected, upgrading a working restore workflow turns it into a Cluster that reports Running with an empty volume. Silent, and it looks like success.

For the upgrade behaviour, the cheapest fix is probably a validation error (or at minimum a Warning event) when metadata.annotations["kubeblocks.io/restore-from-backup"] is set and spec.restore is empty. That converts a silent empty database into something the user sees immediately.

Note for @leon-ape on case 2

I can't narrow the populator regression below alpha.2, because spec.restore doesn't exist before then — alpha.2 vs alpha.3 is the whole available window.

Worth knowing that spec.restore has not completed a restore in either released version that has it:

• v1.2.0-alpha.2 — Restore via Cluster.spec.restore never completes: injected restore-manager sidecar never receives its stop signal (1.2.0-alpha.2) #10749, the kb-populate-* Job restored the data correctly but the injected restore-manager sidecar never exited, so the Job never finished and the Restore stayed in AsDataSource. Closed as duplicate.
• v1.2.0-alpha.3 — this issue, the sidecar no longer hangs but the kb-populate-<uid> PVC is deleted or never created, so the prepareData Job can't be scheduled at all and the Restore stays in Running/PrepareData=False.

The failure moved but the populator path has never produced a restored dataset in a release, as far as I've been able to test. Combined with case 1, that means the only working restore path across all current releases is v1.2.0-alpha.1 plus the annotation that has since been removed, which is why the addon I'm working on is still pinned there.

Happy to re-run the case 2 reproduction against a patched build or a specific commit if that helps — it's a scripted end-to-end scenario (backup, restore, then assert on the restored keys rather than on status.phase), so turning it around is quick.