Skip to content

etcd member actions can keep dialing a stale leader endpoint #3236

Description

@weicao

Problem

The etcd addon injects LEADER_POD_FQDN into a Pod environment variable when the Pod is created. That value is static, but etcd leadership can move while the same Pod and lifecycle action remain in use.

member-join.sh and member-leave.sh then use that stale hostname as their only etcdctl contact. If the former leader is unavailable, repeated lifecycle invocations on the same Pod keep returning member-list-query-failed instead of converging.

This is separate from #3234: #3234 makes member-join replay idempotent once an authoritative member list is available; this issue covers how member actions select a current, authoritative contact after leadership moves.

Reproducible scene

  1. Keep the lifecycle Pod alive with LEADER_POD_FQDN=etcd-0... captured at Pod creation.
  2. Move leadership to etcd-2 and make etcd-0 unavailable to the action.
  3. Invoke member join or member leave again from the same Pod.
  4. Observe that the action contacts only etcd-0, returns rc=1 with phase: member-list-query-failed, and never reaches the healthy current leader.
  5. Invoke it again without recreating the Pod. The static environment value is unchanged, so the same failure repeats.

The focused regression is in:

shellspec --load-path ./shellspec \
  addons/etcd/scripts-ut-spec/member_join_spec.sh \
  addons/etcd/scripts-ut-spec/member_leave_spec.sh

The key cases are uses the action-time local leader instead of stale topology snapshots and uses the action-time local leader and current member list instead of static snapshots.

Expected behavior

  • Run the action on the Pod currently selected by the leader role.
  • Verify that the local endpoint's MemberID equals the authoritative Leader ID before mutation.
  • Build the etcdctl contact list from the action-time authoritative member list, with strict parsing, canonicalization, deduplication, TLS separation, and bounded timeouts.
  • Preserve raw uint64 member-ID comparisons and fail closed on malformed or ambiguous membership.
  • Keep join/leave replay-safe with at most one membership mutation.
  • Return a classified deferral instead of waiting inside the lifecycle script.

Evidence boundary

The local code candidate has focused/full ShellSpec, ShellCheck, Bash syntax, Helm render/lint, BusyBox production-function, and bounded etcdctl timing evidence. It has not been run as Kubernetes product Test and does not establish a release verdict.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions