Describe the bug
In a Redis replication topology (redis + redis-sentinel), if the primary pod AND its PVC are
lost and the InstanceSet recreates the pod faster than sentinel's down-after-milliseconds
(hardcoded to 20000 in redis-register-to-sentinel.sh), sentinel never triggers a failover —
it only logs +reboot master. The start script then queries sentinel
(sentinel get-master-addr-by-name), gets its own address back, and starts the node as
master with an empty data dir. The replica reconnects, run_id changed → full resync
from the empty master → the entire dataset is wiped on all nodes (AOF included, since
replication rewrites the disk).
To Reproduce
- Create a replication Cluster (2 redis replicas + 3 sentinels, addon redis 1.0.4, Redis 8.4.3)
- Write some keys, confirm replication
kubectl delete pvc data-<cluster>-redis-0 --wait=false && kubectl delete pod <cluster>-redis-0
- InstanceSet recreates pod + fresh PVC in ~14s (< down-after 20s)
- Sentinel logs only
+reboot master (no +sdown/+switch-master)
- Reborn pod starts empty as master; replica full-syncs from it (sync_full:1 on master)
- DBSIZE = 0 on all nodes
Expected behavior
A node starting with an empty data dir should not resume as master while replicas with
data exist (e.g. the start script could refuse mastership on empty dir, or trigger a
failover first). At minimum, down-after-milliseconds should be configurable so the
failover can win this race.
Workaround validated
SENTINEL SET <master> down-after-milliseconds 5000 on all sentinels: failover completes
in ~6s, the reborn empty node joins as replica and resyncs from the new primary — no loss.
Environment
- KubeBlocks operator 1.0.2 (CRDs from release-1.0), redis addon chart 1.0.4
- Redis 8.4.3, topology: replication (2 + 3 sentinels)
- Kubernetes RKE2 v1.36.1, storage: local hostpath PVs
Describe the bug
In a Redis replication topology (redis + redis-sentinel), if the primary pod AND its PVC are
lost and the InstanceSet recreates the pod faster than sentinel's down-after-milliseconds
(hardcoded to 20000 in redis-register-to-sentinel.sh), sentinel never triggers a failover —
it only logs
+reboot master. The start script then queries sentinel(
sentinel get-master-addr-by-name), gets its own address back, and starts the node asmaster with an empty data dir. The replica reconnects, run_id changed → full resync
from the empty master → the entire dataset is wiped on all nodes (AOF included, since
replication rewrites the disk).
To Reproduce
kubectl delete pvc data-<cluster>-redis-0 --wait=false && kubectl delete pod <cluster>-redis-0+reboot master(no +sdown/+switch-master)Expected behavior
A node starting with an empty data dir should not resume as master while replicas with
data exist (e.g. the start script could refuse mastership on empty dir, or trigger a
failover first). At minimum, down-after-milliseconds should be configurable so the
failover can win this race.
Workaround validated
SENTINEL SET <master> down-after-milliseconds 5000on all sentinels: failover completesin ~6s, the reborn empty node joins as replica and resyncs from the new primary — no loss.
Environment