Skip to content

[BUG] Redis replication: primary reborn with empty data dir within down-after window resumes as master and wipes replicas via full resync #10753

Description

@leandro-clint

Describe the bug
In a Redis replication topology (redis + redis-sentinel), if the primary pod AND its PVC are
lost and the InstanceSet recreates the pod faster than sentinel's down-after-milliseconds
(hardcoded to 20000 in redis-register-to-sentinel.sh), sentinel never triggers a failover —
it only logs +reboot master. The start script then queries sentinel
(sentinel get-master-addr-by-name), gets its own address back, and starts the node as
master with an empty data dir. The replica reconnects, run_id changed → full resync
from the empty master → the entire dataset is wiped on all nodes (AOF included, since
replication rewrites the disk).

To Reproduce

  1. Create a replication Cluster (2 redis replicas + 3 sentinels, addon redis 1.0.4, Redis 8.4.3)
  2. Write some keys, confirm replication
  3. kubectl delete pvc data-<cluster>-redis-0 --wait=false && kubectl delete pod <cluster>-redis-0
  4. InstanceSet recreates pod + fresh PVC in ~14s (< down-after 20s)
  5. Sentinel logs only +reboot master (no +sdown/+switch-master)
  6. Reborn pod starts empty as master; replica full-syncs from it (sync_full:1 on master)
  7. DBSIZE = 0 on all nodes

Expected behavior
A node starting with an empty data dir should not resume as master while replicas with
data exist (e.g. the start script could refuse mastership on empty dir, or trigger a
failover first). At minimum, down-after-milliseconds should be configurable so the
failover can win this race.

Workaround validated
SENTINEL SET <master> down-after-milliseconds 5000 on all sentinels: failover completes
in ~6s, the reborn empty node joins as replica and resyncs from the new primary — no loss.

Environment

  • KubeBlocks operator 1.0.2 (CRDs from release-1.0), redis addon chart 1.0.4
  • Redis 8.4.3, topology: replication (2 + 3 sentinels)
  • Kubernetes RKE2 v1.36.1, storage: local hostpath PVs

Metadata

Metadata

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions