chore(ci): persist shard/job timeout-vs-cap trending and warn at 90% before a breach

[trek-e]

Internal maintenance work

CI/CD pipeline improvement — no user-facing behavior changes.

Pre-submission checklist

• This does not change user-facing behavior (commands, output, file formats, config)
• I have searched existing issues — this has not already been filed (related-but-distinct prior issues: ci: test (ubuntu-latest, 24) exceeds the 15m cap — full unit suite under coverage no longer fits, next is red #2952, ci: full-test (windows-latest, 22) job timing out at the 15m cap, red-blocking product PRs #869, ci(enhancement): shard the windows full-test lane to durably fix the timeout-cap cancellations #1212, Windows shard 1/3 times out: test sharding is cost-blind and unstable under file insertion #2472 all fixed specific past timeout breaches; chore: a killed test chunk does not name the file that was hanging #4012 added chunk-level hang diagnostics inside one shard's job. None of them add a persisted, accumulating record of shard/job wall-clock vs. timeout-minutes, or a proactive near-cap warning. See "Additional context" below.)

What is the maintenance task?

Add a persisted, accumulating report of every sharded/matrixed CI job's wall-clock time against its timeout-minutes cap, plus an automatic warning when a job reaches ≥90% of that cap — and instrument every matrixed job that currently lacks either.

Type of maintenance

CI/CD pipeline

Current state

Every timeout-minutes cap in this repo is justified by a hand-pasted wall-clock number in a workflow-file comment, taken from one specific past Actions run (e.g. .github/workflows/test.yml:137: "Measured on run 30677442953: shard 1/3 7m12s, 2/3 4m32s, 3/3 3m59s"; test.yml:437: "26m18s (run 32614439702)"). tests/ci-test-job-timeout-budget.test.cjs statically enforces a 1.5x headroom factor against those same hand-typed LANE_COSTS numbers (measuredMinutes/evidence pairs a human edits after reading a log) — it locks the checked-in cap, it does not measure anything live. The file's own comment says so directly: "No unit test can prove a lane still FITS its budget — only a real CI run measures that."

There is no automated record of what a shard/job actually costs over time, and no automatic signal before a cap is breached — the only existing signal is a human noticing a cancelled job in the Actions UI and later hand-editing the comment + LANE_COSTS entry (this is exactly how #2952, #869, #1212, and #2472 were each discovered and fixed, one breach at a time).

Full audit of every matrixed/sharded job in this repo (grep -n "strategy:\|matrix:" .github/workflows/*.yml — only three files match):

Job	Cap	Measured-cost record?	Headroom-gate coverage?
test.yml → test (full + windows lanes, 6 entries)	15m	Yes (comment + LANE_COSTS)	Yes
test.yml → test-full (windows/macos × 3 shards)	45m	Yes (comment + LANE_COSTS)	Yes
mutation.yml → mutate (dynamic per-module Stryker shards)	per-shard, matrix.timeoutMinutes from scripts/mutation-matrix.cjs	Partial — one hand-edited override (frontmatter: 20m) with a comment citing one run; no live measurement	No — not in tests/ci-test-job-timeout-budget.test.cjs's LANE_COSTS at all
install-smoke.yml → smoke (OS matrix)	12m	None	None

#4012's chunk-level NDJSON instrumentation (inside scripts/run-tests.cjs) is a distinct, already-solved problem — it names the in-flight file when a chunk (a sub-shard batch) is killed at its own 600000ms cap. It logs elapsed time per chunk but discards that data on every successful run (scripts/run-tests.cjs:1228-1234, deletes the NDJSON file once a chunk succeeds) and never compares elapsed-vs-cap proactively. It does not touch the job-level (timeout-minutes) report-vs-cap trending this issue is about.

Full research/citations: docs/research/ci-shard-timeout-instrumentation.md (produced via /research against primary GitHub Actions docs + full reads of every file cited above).

Proposed work

Implement the recommendation from docs/research/ci-shard-timeout-instrumentation.md §5:

1. Persisted report — a scheduled workflow (cron, same shape as duplicate-sweep.yml) queries the GitHub REST API (GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs, returns started_at/completed_at per job) for recent runs of test.yml, mutation.yml, and install-smoke.yml, cross-references each job's declared timeout-minutes (parsed from the workflow YAML the same way tests/ci-test-job-timeout-budget.test.cjs already does via js-yaml), and appends one record per job per run to a durable, checked-in history file (JSONL, shaped like tests/test-timings.json's existing role as a machine-generated, versioned data file) on push-to-next runs. Zero changes to the per-shard jobs themselves — this cannot perturb the timings it measures.
2. Near-cap warning — once the report in (1) computes elapsedMs / (timeoutMinutes * 60000) per job, emit a ::warning:: annotation and a $GITHUB_STEP_SUMMARY block (both zero-new-permission, both already idiomatic in this repo — test.yml:79-90, throughout release.yml) whenever any job in the current run is ≥90% of its cap. A durable issue-bot escalation (following duplicate-check.yml/duplicate-sweep.yml's exact shape: actions/github-script + a new scripts/ci-timeout-alert.cjs module + label-gated dedup) for a lane that has been near-cap for multiple consecutive runs (not just one noisy run) may be added if the always-on annotation/summary signal proves insufficient in practice — start with the cheap, always-on layer.
3. Audit remediation — extend tests/ci-test-job-timeout-budget.test.cjs (or its replacement) to cover all four matrixed jobs, not just three: add mutation.yml's mutate job and install-smoke.yml's smoke job to the headroom-factor gate, with a measured-cost record for smoke (currently has none at all).
4. Docs — add a section to docs/TESTING-SUITES.md alongside the existing "CI matrix" (:447-460) and "Chunk packing and the test timing table" (:498-529) sections documenting the new report + warning mechanism.

Explicitly out of scope (maintainer policy calls, not part of this issue): raising any timeout-minutes value, rebalancing test.yml's shards, or trimming shard 1's unsharded aux suites (integration/security/install/slow). This issue instruments and reports; it does not retune the budgets or the packing.

Done when

• A machine-generated, persisted record of each matrixed job's actual wall-clock vs. its timeout-minutes cap exists and accumulates across push-to-next runs, covering test/test-full (test.yml), mutate (mutation.yml), and smoke (install-smoke.yml)
• Every CI run emits a ::warning:: annotation and a $GITHUB_STEP_SUMMARY entry for any job that reaches ≥90% of its timeout-minutes budget, without needing a maintainer to be watching the run live
• install-smoke.yml's smoke job has a measured-cost record and is covered by the headroom-factor gate (currently has neither)
• mutation.yml's mutate job is covered by the headroom-factor gate (currently is not)
• docs/TESTING-SUITES.md documents the new mechanism
• No change to any timeout-minutes value, shard count/composition, or shard-1 contents as part of this issue
• gsd-test green

Area affected

CI/CD

Additional context

Full research and citations: docs/research/ci-shard-timeout-instrumentation.md. Related, already-resolved prior issues (each fixed one specific past breach, none add persisted trending or proactive warning): #2952, #869, #1212, #2472, #4012.

[github-actions]

Branch chore/4036-chore-ci-persist-shard-job-timeout-vs-ca created.

git fetch origin && git checkout chore/4036-chore-ci-persist-shard-job-timeout-vs-ca