Restore is broken on 1.2.0-alpha.3: the annotation path is silently ignored and the spec.restore populator PVC is deleted underneath its own prepareData job

Summary

On 1.2.0-alpha.3 there is no working way to restore a Cluster from a Backup, and both failure modes are silent — the Cluster reaches status.phase: Running with an empty database, so nothing surfaces the problem to the user.

1. kubeblocks.io/restore-from-backup is ignored entirely — no Restore object is created at all. This worked on 1.2.0-alpha.1.
2. Cluster.spec.restore does create a Restore, but its prepareData job can never run because the kb-populate-<uid> PVC it is supposed to mount is deleted (or never created). The Restore stays Running / PrepareData=False indefinitely.

In both cases the database pods start and begin serving an empty dataset while the restore is still incomplete. A user watching kubectl get cluster sees Running and would reasonably conclude the restore succeeded.

This is distinct from #10749 (restore-manager sidecar never receiving its stop signal), which is closed. The sidecar no longer hangs; the failure has moved to the populator PVC lifecycle.

Environment

The Backup itself is healthy and is not implicated:

NAME                  METHOD     REPO                STATUS      TOTAL-SIZE   DURATION
fdb-backup-datafile   datafile   falkordb-e2e-repo   Completed   2724         12s

Case 1 — kubeblocks.io/restore-from-backup is silently ignored

Create a Cluster carrying the annotation:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: fdb-restored
  annotations:
    kubeblocks.io/restore-from-backup: '{"falkordb":{"name":"fdb-backup-datafile","namespace":"<ns>","volumeRestorePolicy":"Parallel"}}'
spec:
  terminationPolicy: WipeOut
  clusterDef: falkordb
  topology: replication
  componentSpecs:
    - name: falkordb
      replicas: 2
      # ... volumeClaimTemplates: data, 1Gi
    - name: falkordb-sent
      replicas: 3

The annotation is present on the created object:

$ kubectl get cluster fdb-restored -n <ns> -o jsonpath='{.metadata.annotations}'
{"kubeblocks.io/restore-from-backup":"{\"falkordb\":{\"name\":\"fdb-backup-datafile\",...}}", ...}

But no restore is ever attempted:

$ kubectl get restores.dataprotection.kubeblocks.io -n <ns>
No resources found in <ns> namespace.

$ kubectl get jobs -n <ns>
No resources found in <ns> namespace.

The Cluster nonetheless goes Running with a correct-looking topology and no data:

$ kubectl get cluster fdb-restored -n <ns>
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
fdb-restored   falkordb             WipeOut              Running   88s

# 1 primary + 1 secondary, replication link up, DBSIZE 0

Expected: either the annotation is honoured, or — if it has been intentionally removed in favour of spec.restore — the Cluster is rejected or a warning event is emitted. Silently dropping a restore request is the problem.

This is a regression: the same annotation on the same addon and the same Backup restores correctly on 1.2.0-alpha.1.

Case 2 — spec.restore hangs because the populator PVC is deleted

Same Backup, using the alpha.3 API instead:

spec:
  restore:
    source:
      apiGroup: dataprotection.kubeblocks.io
      kind: Backup
      name: fdb-backup-datafile
      namespace: <ns>

This is wired up correctly at first — Restore objects are created, and they target only the falkordb data PVC and leave the falkordb-sent PVCs alone, which is the right behaviour:

$ kubectl get restores.dataprotection.kubeblocks.io -n <ns>
NAME                                                      BACKUP                STATUS      DURATION
kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6          fdb-backup-datafile   Running
kb-populate-c821f723-5655-48e0-b6df-b28c39740e8c          fdb-backup-datafile   Running
restore-a9818ceb-c69c-48bc-9f67-7ecc000347f2-post-ready   fdb-backup-datafile   Completed   6s

The two kb-populate-* restores never leave Running:

phase: Running
  prepareData: status=Processing
    "Job/restore-preparedata-abd972dc-fdb-backup-datafile-0" is processing
  conditions:
    CheckBackupRepo    True
    ValidationPassed   True   validate restore spec successfully
    PrepareData        False  processing prepareData stage.

Their jobs can never be scheduled, because each one wants a kb-populate-* PVC that is gone:

$ kubectl describe pod restore-preparedata-abd972dc-fdb-backup-datafile-0-xbhdq -n <ns>
Warning  FailedMount  kubelet
  Unable to attach or mount volumes:
  unmounted volumes=[dp-claim-kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6]
  error processing PVC <ns>/kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6:
  PVC is being deleted

$ kubectl describe pod restore-preparedata-226c33a6-fdb-backup-datafile-0-5v6lf -n <ns>
Warning  FailedScheduling  default-scheduler
  running PreBind plugin "VolumeBinding": binding volumes:
  failed to check provisioning pvc: could not find v1.PersistentVolumeClaim
  "<ns>/kb-populate-c821f723-5655-48e0-b6df-b28c39740e8c"

So one job's populator PVC is terminating while the job tries to mount it, and the other job's populator PVC does not exist at all. The Restore has the PVC in its own spec.prepareDataConfig.volumeClaims[]:

"prepareDataConfig": {
  "volumeClaimRestorePolicy": "Parallel",
  "volumeClaims": [
    { "metadata": { "name": "kb-populate-8bcc2fc0-23cd-407b-9b51-3d08708cd1e6", "namespace": "<ns>" },
      "volumeClaimSpec": { "accessModes": ["ReadWriteOnce"], ... } }
  ]
}

Meanwhile the Cluster reports Running and the database serves an empty dataset:

$ kubectl wait --for=jsonpath='{.status.phase}'=Running cluster/fdb-r2 -n <ns> --timeout=240s
cluster.apps.kubeblocks.io/fdb-r2 condition met

$ kubectl exec fdb-r2-falkordb-0 -c falkordb -- redis-cli DBSIZE
0

Note the two independent problems here: the populator PVC race, and the fact that the workload is allowed to start and serve traffic while prepareData has not completed. Even once the PVC race is fixed, gating pod startup on restore completion would turn a silent empty database into an honest failure.

Expected behaviour

• spec.restore completes, prepareData finishes, and the Cluster comes up with the backed-up data.
• The Cluster does not reach Running while its restore is still in prepareData.
• kubeblocks.io/restore-from-backup either works or is explicitly rejected/warned about rather than silently dropped.

Impact

There is no working restore path on 1.2.0-alpha.3. Because both failures present as a healthy Running Cluster, a restore that silently produces an empty database is very easy to mistake for a successful one — this was only caught by an end-to-end test that asserts on the restored data rather than on status.phase.