Switchover OpsRequest can dispatch the lifecycle action twice when status persistence fails #10649

Problem observed

This issue was found in one focused SQL Server HA switchover run on KubeBlocks main.

The observed sequence was:

1. One Switchover OpsRequest targeted one database component.
2. The first lifecycle-action request returned HTTP 204; the incident packet recorded that request as successful.
3. The controller then failed to persist the progress transition because of an OpsRequest status resource-version conflict.
4. The next reconciliation still saw the item as Pending and sent the same switchover request again about one second later.
5. The addon rejected the second request because the first transition was still unfinished, and the OpsRequest became Failed.
6. The database roles were subsequently observed to have converged.

HTTP 204 alone does not prove that the database action completed, and the available evidence does not prove that the later role convergence was caused by that specific response. It does prove that one OpsRequest dispatched the non-idempotent lifecycle action twice.

Observed runtime scope: one SQL Server case. The original incident does not cover same-component multi-item switchover, the no-candidate branch, or the five-minute non-convergence branch.

Why this violates the expected contract

A single switchover progress item must invoke its non-idempotent lifecycle action at most once. A status-write conflict or lost client response must not turn one logical request into a second database role-transition request.

If the controller still holds a definite success or failure result returned by the lifecycle action, it must keep that result while retrying durable status persistence. Only a later process that finds a committed claim but has no retained call result may classify the outcome as unknown, and it must not dispatch again.

Root cause

Tracing backward from the duplicate external call gives this chain:

1. handleSwitchover() dispatched runtime.Switchover(...) while the progress item was still durably Pending.
2. Only after the external call returned did the controller attempt to write Processing or Failed.
3. A status conflict before that write committed left the API Server state unchanged at Pending.
4. The next reconciliation therefore entered the same dispatch branch again.

The first broken edge is the missing durable pre-dispatch state transition. The repair belongs in the generic Switchover OpsRequest state machine, not in SQL Server or addon-specific retry logic.

Minimal design

1. Before creating any dispatch marker, require a non-empty OpsRequest UID and a direct API Server reader. Missing prerequisites must return with zero lifecycle calls and no server-side status change.
2. Generate a unique claim token bound to the OpsRequest UID, logical component, source instance, optional candidate, and this writer.
3. Use an optimistic-lock status patch to move the existing progress item from Pending to the existing public Processing state and record the exact claim in its message. This adds no CRD field or enum.
4. Read the object directly from the API Server after the patch. Only the writer whose exact token is present may invoke the lifecycle action. A conflict, ambiguous patch response, or foreign token results in zero calls for that writer.
5. After the lifecycle action returns a definite result, repeatedly read the latest API Server object, verify the same token, and retry the optimistic status patch across conflicts or transient errors. A committed patch whose client response was lost is recognized from the exact persisted outcome.
6. Never overwrite a foreign token or a changed claim identity. If a later reconciliation finds a surviving claim after the original process lost the returned result, it observes the explicit candidate when possible or fails closed; it never invokes the lifecycle action again.

Processing is used because supported older controllers already treat it as non-dispatchable. Mixed-version readers still need to understand the retained-claim recovery semantics before this writer is activated in a rollout; code review order starts from this main writer contract, while release activation order keeps readers before the writer.

Out of scope

• no new OpsRequest CRD field or progress enum
• no addon or SQL Server retry workaround
• no claim that the lifecycle action itself is idempotent
• no same-component multi-item identity fix; that remains tracked separately
• no claim that source/unit review proves current-head runtime behavior

Validation plan

• missing API reader and empty OpsRequest UID: API Server resourceVersion/status/message unchanged, lifecycle call count zero
• two stale writers with ambiguous status responses: only the committed unique token dispatches
• known success plus resource-version conflict: fresh read and retry persist success, call count one
• known failure plus transient write error: fresh read and retry persist the original failure, call count one
• committed outcome with lost client response: recognize the persisted result without redispatch
• foreign token or changed identity: do not overwrite and do not redispatch
• process restart with a surviving claim: candidate observation or fail-closed unknown outcome, call count zero
• full Switchover and operations suites, focused race, vet/lint, generated artifacts, and committed-tree diff gate
• exact-image SQL Server runtime validation after the reviewed main candidate is published

Current fix runtime scope: N=0.