vtorc: add topo-based recovery cooldown for ERS/PRS

[sbaker617]

What's this?

Adds a configurable cooldown window (--recovery-cooldown-duration) that prevents VTOrc from performing another cluster-wide recovery (ERS/PRS) on the same shard within a configurable time period. This addresses flapping scenarios where multiple VTOrc instances rapidly execute sequential reparents.

How it works

• After a successful cluster-wide recovery, VTOrc writes a JSON cooldown marker to the topo server metadata at metadata/recovery_cooldown/{keyspace}/{shard}
• Before attempting a new cluster-wide recovery, VTOrc checks for an active cooldown marker — if the marker exists and hasn't expired, the recovery is skipped
• The check happens before LockShard() to avoid unnecessary lock contention across instances
• Marker is stored in the topo global cell, so all VTOrc instances share visibility
• Default is 0 (disabled) for backwards compatibility

What's NOT affected

• Manual vtctldclient ERS/PRS operations — they go through grpcvtctldserver, never touch VTOrc's recovery path
• reparenting-solver — uses vtctldclient gRPC, same unaffected path
• Non-cluster-wide VTOrc recoveries (fixPrimary, fixReplica, etc.) — guarded by isClusterWideRecovery()

New config

Flag	Default	Description
--recovery-cooldown-duration	0 (disabled)	Duration after a VTOrc-initiated ERS/PRS during which VTOrc will not attempt another ERS/PRS on the same shard

New metric

• RecoveriesCooldownSkipped (labels: RecoveryType, Keyspace, Shard) — counts skipped recoveries due to active cooldown

Design note: file structure for fork maintainability

The first commit inlines everything into topology_recovery.go. The second commit restructures to minimize the diff against upstream files — all cooldown logic and tests live in new standalone files (topology_recovery_cooldown.go, topology_recovery_cooldown_test.go) that we fully own. The only edits to upstream files are:

• topology_recovery.go: ~19 lines (two call sites that invoke the cooldown check/write)
• config.go: ~20 lines (flag definition, registration, getter)
• metadata.go: ~13 lines (GetSingleMetadata wrapper)

This way upstream backports only risk conflicts at the small call-site insertions, not the full implementation. Reviewer can compare the two commits to see both approaches.

---

Most of this was written by Claude Code — I just provided direction.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.80%. Comparing base (8ab2b14) to head (7a60a5e).

Files with missing lines	Patch %	Lines
go/vt/vtorc/logic/topology_recovery_cooldown.go	0.00%	36 Missing ⚠️
go/vt/vtorc/logic/topology_recovery.go	0.00%	18 Missing ⚠️
go/vt/topo/metadata.go	0.00%	7 Missing ⚠️
go/vt/vtorc/config/config.go	0.00%	6 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           slack-22.0     #819      +/-   ##
==============================================
- Coverage       69.80%   69.80%   -0.01%     
==============================================
  Files            1606     1607       +1     
  Lines          214099   214166      +67     
==============================================
+ Hits           149454   149490      +36     
- Misses          64645    64676      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

• Push additional commits to the associated branch.
• Remove the stale label.
• Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

[github-actions]

This PR was closed because it has been stale for 7 days with no activity.