fix(health): bound health checks that perform I/O

[kimsondrup]

Overview

Health checks that perform I/O can be blocked indefinitely by the very dependency
they are checking. When that happens the check does not fail — it stops returning,
and the scheduler goes on serving the last result it recorded, which is a passing
one. Dex then reports itself healthy for the whole outage.

This wraps the check so that an execution which overruns is reported as a failure,
and applies it to the storage check.

What this PR does / motivation

We hit this in production. A dex instance became unable to serve any storage
operation and stayed that way for over 13 hours, reporting healthy throughout:
/healthz/ready answered 200 in 8ms and the load balancer kept it in rotation.
The storage check is a real round trip, so it was blocked by the same condition it
exists to detect — it did not fail, it stopped returning, and a check that never
returns records nothing.

The triggering bug is separate and fixed separately. This PR is about the health
check being unable to report it, or the next one.

Why the existing timeout option is not sufficient

go-sundheit records a result when a check returns. checkTask.execute invokes
the check synchronously, so a check that never returns records nothing and the
previously recorded (passing) result stands.

Setting gosundheit.ExecutionTimeout does not address this: it only cancels the
context, and cancellation is cooperative. A check blocked where cancellation is
not observed — waiting on a database/sql connection pool, inside a driver that
takes no context, in a syscall, in cgo — never returns regardless. In our case the
check was parked in a pool wait, and the SQL storage layer does not propagate
context to database/sql at all, so there was nothing to cancel.

What this changes

storage.NewTimeoutHealthCheckFunc wraps a check function so that:

• the check runs on its own goroutine, and the caller stops waiting once it
  overruns, reporting the overrun as a failure;
• the context is canceled on overrun, so checks that do observe cancellation
  unblock promptly;
• executions are single-flight.

Single-flight matters because Go cannot stop a goroutine: one that is blocked keeps
running and completes only if the dependency recovers. Starting a fresh one per
scheduler tick would accumulate one goroutine per tick for the length of the outage
(thousands, over hours), each still holding whatever it had acquired. Instead,
while an execution is outstanding, later ones report the failure without starting
another.

One consequence, deliberate: an execution that never returns holds the slot for
the life of the process, so the check reports a failure from then on
even if the dependency later recovers. An execution that cannot be completed
attests to nothing, and a process holding resources it may never release is not
one to report as healthy. Recovery is by restart, which is the outcome such a
process needs anyway. This is documented on the exported function and pinned by a
test.

This bounds the cost of an outage and makes it visible. It does not make the
process any healthier, and is not a substitute for fixing the underlying faults.

Applied to the storage check in cmd/dex/serve.go — currently the only registered
check, and the only one that performs I/O. (/healthz/live is static;
server.go's /healthz reads the same cached verdict and is fixed by the same
change.)

Tests

storage/health_test.go (12 tests) — a check that never returns is reported as a
failure; single-flight is enforced; the check recovers once the outstanding
execution returns; a stuck execution keeps the check failing and is never
restarted; the goroutine releases the slot before delivering its result; the
context is canceled on overrun; caller cancellation releases the caller; a stale
result is never handed to a later caller; details and errors pass through; defaults
are applied; concurrent executions under -race.

storage/health_scheduler_test.go — drives the real go-sundheit scheduler and
asserts it goes unhealthy when the check stops returning. Pointed at the unwrapped
check this test fails, reproducing the symptom above: the scheduler keeps reporting
healthy.

Statement coverage of the wrapper is 100%.

Special notes for reviewers

• Defaults. DefaultHealthCheckTimeout is 10s against the existing 15s
  execution period, so a stalled execution is reported before the next is due. It is
  deliberately generous: it bounds a stall, it does not police latency. It is not
  currently configurable — happy to expose it in config if you would prefer that
  before merge, particularly for larger SQL or etcd deployments where 10s might be
  tight under load.

• InitiallyPassing(true) is unchanged. It means dex reports healthy for up to
  one period before the first check completes. Arguably wrong for a readiness probe,
  but it is existing behavior and changing it affects every deployment, so it seemed
  out of scope here.

• Placement. The wrapper makes no assumptions about the check it is given, so
  pkg/ may be a better home than storage. It is in storage because that is
  where the only check that needs it lives. Happy to move it.

• Wrapping is applied at the registration site, not inside the check, so a future
  check that performs I/O must be wrapped explicitly. Wrapping inside RegisterCheck
  is not possible without vendoring go-sundheit. With one check this is fine; at
  two it is probably worth a small register-and-wrap helper.

• Panics are not recovered. go-sundheit has no recover(), so a panicking
  check crashes the process both before and after this change. Behavior is unchanged;
  converting a panic to a check failure would be a reasonable follow-up but is a
  behavior change beyond "do not hang".

• Upstream alternative. The cleanest long-term home for this is go-sundheit
  itself — checkTask.execute could run the check on a goroutine and record a
  timeout result on overrun, fixing it for every consumer. Happy to raise that
  upstream. This wrapper would still be wanted in the interim, and dex would keep
  control of the single-flight policy.

• The behavior change to be aware of: a check that correctly starts failing will
  now cycle a container configured with a health-gated restart policy, where
  previously the outage was silent. That is the intent, but it is a visible
  difference for operators.