Skip to content

[BUG] backup-policy-driver-controller failed to reconcile: sharding components shard not found #10769

Description

@JashBook

Describe the bug

When a Cluster uses sharding with a shared system account (defined in ShardingDefinition with shared: true), the component controller keeps requeuing with:

wait for referenced account secret <namespace>/<cluster>-<shard>-default revision

  Warning  ReconcileBackupPolicyFail  4m22s (x17 over 9m50s)  backup-policy-driver-controller  failed to reconcile: sharding components shard not found

The shard Component never progresses past the ProvisioningStarted condition and has no ApplyResources condition, so the cluster is stuck in Creating phase.

Root cause: clusterShardingAccountTransformer.rewriteSystemAccount updates transCtx.shardings[i].Template.SystemAccounts and transCtx.shardingComps[shardingName] with the correct SecretRef + SecretRefRevision, but it does not update transCtx.shardingCompsWithTpl.

clusterComponentTransformer.buildComps uses transCtx.shardingCompsWithTpl as the source of truth when creating Component CRs. Since shardingCompsWithTpl is populated by clusterNormalizationTransformer (which runs before clusterShardingAccountTransformer), the Component spec ends up with an empty secretRefRevision.

The component controller then checks in componentAccountTransformer.getPasswordFromSecret:

if revision, ok := secret.Annotations[constant.SecretRevisionAnnotationKey]; ok && revision != account.SecretRefRevision {
    return nil, ctrlutil.NewRequeueError(time.Second,
        fmt.Sprintf("wait for referenced account secret %s/%s revision %s", ...))
}

The shared account Secret (created by the cluster controller) has a non-empty revision annotation, but account.SecretRefRevision is empty ("") on the Component spec, so the condition revision != "" is true and the component requeues forever.

To Reproduce

  1. Deploy KubeBlocks 1.1.0-beta.9 (or any build with the SecretRefRevision field, introduced in PR fix(apps): reconcile shared account secret rotations #10743)
  2. Install an addon that defines a sharding topology with shared system accounts (e.g. redis cluster topology, where ShardingDefinition has systemAccounts[0].shared: true)
  3. Create a Cluster CR with sharding topology (e.g. redis cluster mode, apps.kubeblocks.io/v1)
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: rediscl-cfwziv
  namespace: default
spec:
  clusterDef: redis
  topology: cluster
  terminationPolicy: DoNotTerminate
  services:
  - componentSelector: shard
    name: secondary
    serviceName:  shard-secondary
    roleSelector: secondary
    spec:
      ports:
        - name: redis-cluster
          port: 6379
          protocol: TCP
          targetPort: redis-cluster
  shardings:
  - name: shard
    shards: 3
    shardingDef: redis-cluster
    template:
      name: redis
      componentDef: redis-cluster-8-1.1.0-alpha.0
      serviceVersion: 8.0.1
      replicas: 2
      services:
      - name: redis-advertised
        serviceType: NodePort
        podService: true
      systemAccounts:
      - disabled: false
        name: default
        secretRef:
          name: rediscl-cfwziv-shard-default
          namespace: default
          password: password
      resources:
        limits:
          cpu: 100m
          memory: 0.5Gi
        requests:
          cpu: 100m
          memory: 0.5Gi
      persistentVolumeClaimRetentionPolicy:
        whenDeleted: Delete
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
               requests:
                 storage: 20Gi
  1. Observe:
    • Cluster shows ApplyResourcesSucceed: True and Phase: Creating
    • Shard Components are created but stay with only ProvisioningStarted condition, no ApplyResources condition
    • Component controller logs repeat: wait for referenced account secret <cluster>-shard-default revision
kubectl get cluster rediscl-cfwziv
NAME             CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS   AGE
rediscl-cfwziv   redis                DoNotTerminate                9m39s
➜  ~ 
➜  ~ kubectl describe cluster rediscl-cfwziv 
Name:         rediscl-cfwziv
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=redis
Annotations:  kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1
API Version:  apps.kubeblocks.io/v1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2026-08-12T10:44:28Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        1
  Resource Version:  266082682
  UID:               e5fbf384-475e-486c-aa67-8e23abc70c5d
Spec:
  Cluster Def:  redis
  Services:
    Component Selector:  shard
    Name:                secondary
    Role Selector:       secondary
    Service Name:        shard-secondary
    Spec:
      Ports:
        Name:         redis-cluster
        Port:         6379
        Protocol:     TCP
        Target Port:  redis-cluster
  Shardings:
    Name:          shard
    Sharding Def:  redis-cluster
    Shards:        3
    Template:
      Component Def:          redis-cluster-8-1.1.0-alpha.0
      Flat Instance Ordinal:  false
      Name:                   redis
      Persistent Volume Claim Retention Policy:
        When Deleted:     Delete
      Pod Update Policy:  PreferInPlace
      Replicas:           2
      Resources:
        Limits:
          Cpu:     100m
          Memory:  0.5Gi
        Requests:
          Cpu:          100m
          Memory:       0.5Gi
      Service Version:  8.0.1
      Services:
        Name:          redis-advertised
        Pod Service:   true
        Service Type:  NodePort
      System Accounts:
        Disabled:  false
        Name:      default
        Secret Ref:
          Name:       rediscl-cfwziv-shard-default
          Namespace:  default
          Password:   password
      Volume Claim Templates:
        Name:  data
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:  20Gi
  Termination Policy:   DoNotTerminate
  Topology:             cluster
Status:
  Conditions:
    Last Transition Time:  2026-08-12T10:44:28Z
    Message:               The operator has started the provisioning of Cluster: rediscl-cfwziv
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2026-08-12T10:44:28Z
    Message:               secrets "rediscl-cfwziv-shard-default" not found
    Reason:                ApplyResourcesFailed
    Status:                False
    Type:                  ApplyResources
Events:
  Type     Reason                     Age                     From                             Message
  ----     ------                     ----                    ----                             -------
  Normal   PreCheckSucceed            9m50s                   cluster-controller               The operator has started the provisioning of Cluster: rediscl-cfwziv
  Warning  Warning                    9m40s (x12 over 9m50s)  cluster-controller               secrets "rediscl-cfwziv-shard-default" not found
  Warning  ApplyResourcesFailed       4m22s (x17 over 9m50s)  cluster-controller               secrets "rediscl-cfwziv-shard-default" not found
  Warning  ReconcileBackupPolicyFail  4m22s (x17 over 9m50s)  backup-policy-driver-controller  failed to reconcile: sharding components shard not found

logs KB

kubectl logs -l app.kubernetes.io/name=kubeblocks --namespace kb-system --tail 20
2026-08-12T10:49:56.186Z	INFO	sharding components shard not found	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"rediscl-cfwziv","namespace":"default"}, "namespace": "default", "name": "rediscl-cfwziv", "reconcileID": "80ef649a-af9d-4772-8c02-3aa063a8807e", "cluster": {"name":"rediscl-cfwziv","namespace":"default"}}
2026-08-12T10:49:56.186Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"rediscl-cfwziv","namespace":"default"}, "namespace": "default", "name": "rediscl-cfwziv", "reconcileID": "80ef649a-af9d-4772-8c02-3aa063a8807e", "error": "sharding components shard not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2026-08-12T10:55:23.876Z	INFO	sharding components shard not found	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"rediscl-cfwziv","namespace":"default"}, "namespace": "default", "name": "rediscl-cfwziv", "reconcileID": "6f0889eb-5b78-43d7-b680-82fc8243f927", "cluster": {"name":"rediscl-cfwziv","namespace":"default"}}
2026-08-12T10:55:23.876Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"rediscl-cfwziv","namespace":"default"}, "namespace": "default", "name": "rediscl-cfwziv", "reconcileID": "6f0889eb-5b78-43d7-b680-82fc8243f927", "error": "sharding components shard not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227

Expected behavior

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions