Skip to content

Commit bac5f8a

Browse files
HarperHarper
authored andcommitted
feat: v0.8.0 — L9 CadenceGuard + panic ingest + CircuitBreaker
Ports Layer 9 from Harper's internal fork into the open-source mono file. L9 addresses the kernel-panic class that lives below the SIGABRT handler — no Python signal handler can fire because the machine has already rebooted by the time userspace sees anything. The only defence left is to avoid the panic trigger in the first place. metal_guard.py: - CadenceGuard / CadenceViolation / require_cadence_clear — refuse back-to-back loads of the same model within a configurable window (default 180 s). Persisted to ~/.cache/metal-guard/cadence.json so the mark survives subprocess spawn and kernel panic. - parse_panic_reports / ingest_panics_jsonl — scan /Library/Logs/DiagnosticReports/*.panic (+ Retired/) and dedupe- append into ~/.cache/metal-guard/panics.jsonl. - CircuitBreaker / MLXCooldownActive — refuse new workers after ≥ 2 panics in the trailing 1 h (state persisted so the cooldown outlives the current process). - detect_panic_signature + _KERNEL_PANIC_SIGNATURES — classify a panic log into prepare_count_underflow / pending_memory_set / ctxstore_timeout / metal_oom. - kv_cache_clear_on_pressure — ready-made on_pressure callback for start_kv_cache_monitor. - MetalGuard.detect_hardware() now also returns gpu_driver_version (IOGPUFamily kext — mlx#3186 forensic context). tests/test_l9_*.py: 31 new tests (cadence 8, breaker 9, ingest 8, signature 6). Full suite: 157 passed (126 existing + 31 new). README overhaul: collapsed the v0.3–v0.7 per-version narrative sections into a single L1–L9 + R-series feature list with one-line API descriptions and tables. History and rationale now live in CHANGELOG only. New "Limitations" section makes the userspace-only scope of MetalGuard explicit — it lowers panic rate and contains blast radius but cannot fix the kext bug itself. All three language versions (en / zh-TW / ja) updated. README length: 914 → 340 lines.
1 parent 399b7e6 commit bac5f8a

10 files changed

Lines changed: 1816 additions & 2052 deletions

CHANGELOG.md

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,123 @@ All notable changes to **metal-guard** are documented here.
55
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
66
and this project adheres to [Semantic Versioning](https://semver.org/).
77

8+
## [0.8.0] — 2026-04-17
9+
10+
Minor release porting **Layer 9 (L9)** from Harper's internal fork
11+
into the open-source distribution. L9 was written in response to a
12+
production kernel panic on 2026-04-16 23:33:27 — `IOGPUMemory.cpp:492
13+
"completeMemory() prepare count underflow"` — that fired during the
14+
*first* `generate()` after a freshly-loaded subprocess worker. The
15+
L6 SIGABRT handler could not catch it: the panic lived at kernel
16+
level, before any user-space signal. The only defence left was to
17+
stop doing back-to-back loads in the first place.
18+
19+
After 24 hours of real-world exposure, the baseline panic rate on
20+
Harper's box dropped from ~1.4/day to zero — the first
21+
panic-free 24 h window in the post-panic sample. L9 is now stable
22+
enough to ship outside the private fork.
23+
24+
### Added
25+
26+
- **`CadenceGuard`** (L9 primary defence). Per-model load-timestamp
27+
store with configurable minimum interval (default **180 s**).
28+
Persisted to `~/.cache/metal-guard/cadence.json` via atomic
29+
write (`os.replace`), so the mark survives subprocess spawn,
30+
process kill, and kernel panic. A worker can check-and-mark in
31+
one call via the `require_cadence_clear(model_id)` helper; on
32+
violation it raises `CadenceViolation` with `.model_id`,
33+
`.last_ts`, `.min_interval`, and `.delta` for structured
34+
handling. Stale entries (> 4 h) are GC'd on next mark.
35+
36+
- **`CadenceViolation`** exception. Callers inside the worker
37+
subprocess are expected to `sys.exit(2)` on catch so the parent
38+
runner propagates it as a normal worker error rather than a
39+
mysterious crash.
40+
41+
- **Panic ingest** — `parse_panic_reports(directory, *,
42+
since_ts=None)` scans `/Library/Logs/DiagnosticReports/` (and
43+
its `Retired/` subdir) for `*.panic` files, classifies each via
44+
`detect_panic_signature()`, and returns a list of `dict`
45+
records with `ts` / `signature` / `explanation` / `pid` /
46+
`source_file`. Timestamp comes from the embedded `Calendar:
47+
0x<sec> 0x<usec>` field; falls back to `os.path.getmtime` when
48+
absent. **Never raises** — returns `[]` if the directory is
49+
unreadable (modern macOS requires admin privileges for
50+
`/Library/Logs/DiagnosticReports/`).
51+
52+
- **`ingest_panics_jsonl(*, report_dir=None, jsonl_path=None)`**
53+
appends new panic records into a dedupe'd JSONL archive
54+
(default `~/.cache/metal-guard/panics.jsonl`). Dedupes by both
55+
`source_file` and `(ts_bucket, pid)` event key to handle the
56+
macOS quirk where a single panic produces both
57+
`.contents.panic` and `panic-full-...` copies. Idempotent:
58+
returns the number of new records (0 once caught up).
59+
Best-effort writes — panic archival must never itself crash the
60+
caller.
61+
62+
- **`CircuitBreaker`** (L9 secondary defence). Reads the JSONL
63+
archive and refuses new workers when **≥2 panics within the
64+
trailing 1 h** (both thresholds configurable). A trip persists
65+
via `~/.cache/metal-guard/breaker.json` so the cooldown survives
66+
process restart. `check()` raises `MLXCooldownActive` which
67+
carries `panic_count`, `window_sec`, `cooldown_until`, and
68+
`remaining_sec` so the caller can surface an HTTP 503 / task
69+
decline / CLI exit rather than retry-and-panic. `status()`
70+
returns a dashboard-safe snapshot; `clear()` is the operator
71+
override.
72+
73+
- **`detect_panic_signature(text)`** classifies a kernel-panic log
74+
snippet into one of four signatures:
75+
`prepare_count_underflow` (IOGPUMemory.cpp:492 — mlx-lm#883 /
76+
#1015), `pending_memory_set` (IOGPUGroupMemory.cpp:219 —
77+
mlx#3346), `ctxstore_timeout` (mlx#3267), and a `metal_oom`
78+
fallback. Returns `(None, None)` for panics that do not match
79+
any known MLX-related signature — callers can route those to a
80+
generic bucket.
81+
82+
- **`kv_cache_clear_on_pressure(available_gb,
83+
growth_rate_gb_per_min)`** — ready-made callback for
84+
`MetalGuard.start_kv_cache_monitor(on_pressure=...)`. Calls
85+
`mx.clear_cache()` and logs the trigger. No-op when MLX is not
86+
importable.
87+
88+
- **`MetalGuard.detect_hardware()`** now also returns
89+
`gpu_driver_version` (the IOGPUFamily kext bundle version read
90+
via `kextstat`/`ioreg`). Panic reports on mlx#3186 pin the
91+
fault to this specific kext, so recording the driver revision
92+
at startup adds forensic context for future crash correlation.
93+
Value is `None` if the kext reader fails.
94+
95+
### Tests
96+
97+
- 31 new tests under `tests/test_l9_*.py` covering CadenceGuard
98+
(8), CircuitBreaker (9), panic ingest (8), and signature
99+
detection (6). Full suite: **157 passed** (126 existing + 31
100+
new) on Python 3.14.
101+
102+
### Rationale
103+
104+
v0.7.1 closed the prefill-allocation gap (R4 auto-wired into
105+
`_worker_main`). But a workload that stayed under the R4 ceiling
106+
could still panic if it loaded a new model while the previous
107+
one's IOGPU accounting hadn't fully drained. The Harper panic on
108+
2026-04-16 reproduced exactly that path: two 4-bit loads within
109+
~40 s. L9 enforces ≥180 s cadence per-model and trips a 1 h
110+
cooldown after any 2-panic cluster in a rolling hour. Empirically
111+
this window catches catastrophic clusters without punishing the
112+
steady-state background rate.
113+
114+
### Path defaults
115+
116+
Open-source defaults use `~/.cache/metal-guard/` for all L9
117+
artifacts:
118+
119+
- `~/.cache/metal-guard/cadence.json` — CadenceGuard timestamps
120+
- `~/.cache/metal-guard/panics.jsonl` — panic archive
121+
- `~/.cache/metal-guard/breaker.json` — CircuitBreaker state
122+
123+
All three are overridable via constructor / keyword arguments.
124+
8125
## [0.7.1] — 2026-04-16
9126

10127
Patch release wiring R4 (`require_prefill_fit`) into the actual call

0 commit comments

Comments
 (0)