Sharding + TLS: each shard gets its own self-signed CA, so peer TLS between shards can never validate

[dudizimber]

Describe the bug

When TLS is enabled on a Cluster that uses a sharding, KubeBlocks issues a
separate self-signed CA per Component. Every shard of a sharding is its own
Component, so each shard receives a certificate signed by a CA that none of its
peers trusts.

Any engine whose cluster bus performs certificate chain validation between nodes
(Redis/Valkey/FalkorDB tls-cluster yes, and by extension any sharded engine
doing mTLS between shards) therefore cannot form a cluster at all when
issuer: {name: KubeBlocks} is used.

Version

KubeBlocks v1.2.0-alpha.1 (also reproduces on v1.2.0-alpha.3).

To Reproduce

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: fdb-shtls
spec:
  terminationPolicy: WipeOut
  clusterDef: falkordb
  topology: cluster
  shardings:
    - name: shard
      shards: 3
      template:
        name: falkordb-cluster
        replicas: 1
        tls: true
        issuer:
          name: KubeBlocks
        ...

Then compare the CA that each shard received:

$ for c in fdb-shtls-shard-4xv fdb-shtls-shard-c96 fdb-shtls-shard-gc5; do
    kubectl get secret "$c-tls-certs" -o jsonpath='{.data.ca\.crt}' \
      | base64 -d | openssl x509 -noout -fingerprint
  done
SHA1 Fingerprint=17:26:3B:9F:34:CE:AC:37:...
SHA1 Fingerprint=98:EB:BD:AB:50:CA:19:7A:...
SHA1 Fingerprint=1D:7C:5F:8A:38:A1:48:10:...

Three distinct CAs, all with issuer=CN=KubeBlocks.

Expected behavior

All Components that belong to the same sharding (and arguably all Components of
the same Cluster) should be issued certificates from one CA, so that
peer-to-peer TLS between them can validate.

Actual behavior

Each shard trusts only its own CA. The engine's cluster bus rejects every peer:

[falkordb] - Connection with Node 8372627c19fdb381504bfd9e303ca04f8ae49d1d
  at 10.42.0.101:16379 failed: error:0A000086:SSL routines::certificate verify failed

The nodes never leave the handshake state:

127.0.0.1:6379> CLUSTER NODES
... master - 0 0 0 handshake - 0 0 0 disconnected
127.0.0.1:6379> CLUSTER INFO
cluster_state:fail
cluster_known_nodes:2

redis-cli --tls --insecure --cluster create then blocks forever in "Waiting
for the cluster to join", the postProvision lifecycle action is killed by
kbagent at its 30s budget (cost: 30003, result: timedOut), and the Component
stays in Creating:

Available=False/PhaseCheckFail
Progressing=True/PostProvision
Healthy=False/WorkloadNotReady

Proof that the CA is the only blocker

Copying shard c96's ca.crt into shard 4xv and connecting to c96 with it:

$ openssl s_client -connect 10.42.0.101:6379 -CAfile /tmp/peer-ca.crt </dev/null
...
Verification: OK
Verify return code: 0 (ok)

So the certificates themselves are fine — the cluster bus does chain validation
only, with no hostname/IP verification. The single reason the handshake fails is
that the peers do not share a trust root. Nothing else about the generated
certificates needs to change.

Workaround

Switch to issuer: {name: UserProvided} and supply one Secret containing a CA
shared by every shard. With that change the identical test goes from a 20-minute
timeout to passing in 163s.

Impact

issuer: KubeBlocks is unusable for any sharded engine that authenticates
peers over TLS. It is the default issuer, so this is a silent trap: the Cluster
simply never becomes ready and the only diagnostic is inside the engine's own
debug log.

[shanshanying]

Hi @dudizimber

KubeBlocks supports shared TLS credentials across all shards through ShardingDefinition.spec.tls.shared since v1.1.0-alpha.0

For this sharding definition, we can configure:

apiVersion: apps.kubeblocks.io/v1
kind: ShardingDefinition
spec:
  tls:
    shared: true

and then set reference to this sharding def when creating cluster

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  shardings:
    - name: shard
      shardingDef: <your-sharding-def-name>
      template:
        tls: true
...

Hope it helps.

[dudizimber]

Thanks @shanshanying — that field is exactly the right shape for this, so I wired it up. It does not take effect: ShardingDefinition.spec.tls.shared is accepted by the CRD and then silently ignored, because the transformer that implements it is never added to the cluster reconciler's transformer chain.

What I ran

Set it on the addon's ShardingDefinition:

apiVersion: apps.kubeblocks.io/v1
kind: ShardingDefinition
metadata:
  name: falkordb-cluster
spec:
  tls:
    shared: true

and switched the cluster to a plain KubeBlocks issuer, exactly as suggested:

  shardings:
    - name: shard
      shardingDef: falkordb-cluster
      shards: 3
      template:
        tls: true
        issuer:
          name: KubeBlocks

The ShardingDefinition accepts and stores it, and stays healthy:

$ kubectl get shardingdefinition falkordb-cluster -o jsonpath='{.status.phase} {.spec.tls}'
Available {"shared":true}

What happened

No sharding-scoped secret is ever created. reconcileShardingTLS should produce <cluster>-<sharding>-tls, i.e. fdb-shtls-shard-tls:

$ kubectl get secret -n "$NS" --no-headers -o custom-columns=N:.metadata.name
fdb-shtls-shard-8zb-account-default
fdb-shtls-shard-8zb-tls-certs
fdb-shtls-shard-c84-account-default
fdb-shtls-shard-c84-tls-certs
fdb-shtls-shard-default
fdb-shtls-shard-t95-account-default
fdb-shtls-shard-t95-tls-certs

rewriteTLSConfig should have rewritten the sharding template's issuer to UserProvided pointing at that secret. The stored cluster still has the original:

$ kubectl get cluster fdb-shtls -o jsonpath='{.spec.shardings[0].template.issuer}'
{"name":"KubeBlocks"}

And the three shards still get three different CAs — the original bug, unchanged:

fdb-shtls-shard-8zb-tls-certs  f727d820b0c8ffda656611f8e933ac83
fdb-shtls-shard-c84-tls-certs  4e61b8b660bcbf2eb1b21eb9deb4e9a3
fdb-shtls-shard-t95-tls-certs  204cc5cd9498f20b9ed4327e2693405e

(md5 of each secret's ca.crt.) The cluster never forms; every shard fails its post-provision action, which is where redis-cli --cluster create runs.

Root cause

controllers/apps/cluster/transformer_cluster_sharding_tls.go exists and is unit-tested, but clusterShardingTLSTransformer is never passed to AddTransformer. Its sibling is, which is what makes the gap easy to miss — in cluster_controller.go:

        // handle cluster shared account
        &clusterShardingAccountTransformer{},
        // handle cluster services
        &clusterServiceTransformer{},

clusterShardingAccountTransformer is registered, clusterShardingTLSTransformer is not. A repo-wide search for the symbol finds it only in its own file and in transformer_cluster_sharding_test.go, so the unit tests pass while nothing ever calls it during reconcile.

Affected versions

I checked the transformer file and its registration at each tag:

ref	transformer_cluster_sharding_tls.go	registered in cluster_controller.go
v1.1.0-alpha.4	present	no
v1.1.0-beta.0	present	no
v1.1.0-beta.9	present	no
v1.2.0-alpha.0	present	no
v1.2.0-alpha.1	present	no
v1.2.0-alpha.2	present	no
v1.2.0-alpha.3	present	no
main	present	no

Scanning the last 30 commits to cluster_controller.go turns up no revision where it was registered, so as far as I can tell spec.tls.shared has never worked, rather than having regressed. The runtime reproduction above is on v1.2.0-alpha.1.

So the original report stands, and the fix looks like a one-line registration. Happy to open a PR adding it to the chain if that's useful.

One side note for whoever picks this up

ShardingDefinition.spec is immutable and hashed wholesale into apps.kubeblocks.io/immutable-hash. Unlike ComponentDefinitions, ShardingDefinitions in the addon charts are not version-suffixed, so helm upgrade patches the same object in place. Adding spec.tls to an existing install therefore fails immutableCheck and parks the definition in Unavailable, taking every cluster that references it with it, unless apps.kubeblocks.io/skip-immutable-check: "true" is on the object first. Most of the addon charts put that annotation on their ComponentDefinitions but not on their ShardingDefinitions (addons/redis/templates/shardingdefinition.yaml includes redis.apiVersion, which only emits crd-api-version). Worth flagging in the release note when this is fixed, otherwise the documented workaround breaks upgrades.

[shanshanying]

HI @dudizimber

thanks for the msg in detail.

To workaround , maybe you can use UserProvided as an alternative (and you dont need a ShardingDef in this case)

      template:
        tls: true
        issuer:
          name: UserProvided
          secretRef:
            name: fdb-shared-tls
            ca: ca.crt
            cert: tls.crt
            key: tls.key

[dudizimber]

HI @dudizimber

thanks for the msg in detail.

To workaround , maybe you can use UserProvided as an alternative (and you dont need a ShardingDef in this case)

  template:
    tls: true
    issuer:
      name: UserProvided
      secretRef:
        name: fdb-shared-tls
        ca: ca.crt
        cert: tls.crt
        key: tls.key

Hi @shanshanying, yes, that'd definitely work as a workaround, but the fix proposed doesn't seem too complicated.