Skip to content

Commit 05387d2

Browse files
HarperbotHarperbot
authored
Merge pull request #2 from Harperbot/v0.9.0-panic-11-hardening
v0.9.0 — cross-model cadence + gemma-4 floor + KNOWN_PANIC_MODELS
2 parents bac5f8a + cd8678e commit 05387d2

10 files changed

Lines changed: 1738 additions & 11 deletions

‎.github/workflows/ci.yml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
# Primary job: library must pass tests on plain Python without MLX installed.
11+
# All code paths degrade gracefully when `mlx.core` is unimportable, so this
12+
# matrix is the honest portability signal. Covers Python 3.11 / 3.12 / 3.13
13+
# on both Linux and macOS.
14+
test-without-mlx:
15+
name: test (no-mlx, py${{ matrix.python }}, ${{ matrix.os }})
16+
runs-on: ${{ matrix.os }}
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
os: [ubuntu-latest, macos-latest]
21+
python: ["3.11", "3.12", "3.13"]
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- name: Set up Python ${{ matrix.python }}
26+
uses: actions/setup-python@v5
27+
with:
28+
python-version: ${{ matrix.python }}
29+
30+
- name: Install metal-guard (no MLX extra)
31+
run: |
32+
python -m pip install --upgrade pip
33+
python -m pip install -e ".[test]" || python -m pip install -e . pytest
34+
35+
- name: Import smoke test
36+
run: python -c "import metal_guard; print(metal_guard.__name__)"
37+
38+
- name: Run pytest
39+
run: python -m pytest tests/ -q
40+
41+
# Apple Silicon job: install the [mlx] extra and run the real MLX-backed
42+
# paths. MLX only ships wheels for Apple Silicon, so this is macos-14+
43+
# (arm64 runner) only. Allowed to fail on runner unavailability — Linux
44+
# matrix is the gating signal.
45+
test-with-mlx:
46+
name: test (with-mlx, py${{ matrix.python }}, macos-14)
47+
runs-on: macos-14
48+
continue-on-error: true
49+
strategy:
50+
fail-fast: false
51+
matrix:
52+
python: ["3.12"]
53+
steps:
54+
- uses: actions/checkout@v4
55+
56+
- name: Set up Python ${{ matrix.python }}
57+
uses: actions/setup-python@v5
58+
with:
59+
python-version: ${{ matrix.python }}
60+
61+
- name: Install metal-guard with MLX extra
62+
run: |
63+
python -m pip install --upgrade pip
64+
python -m pip install -e ".[mlx]" pytest
65+
66+
- name: Confirm MLX installed
67+
run: python -c "import mlx.core as mx; print('mlx', mx.__version__)"
68+
69+
- name: Run pytest
70+
run: python -m pytest tests/ -q
71+
72+
lint:
73+
name: lint (ruff, advisory)
74+
runs-on: ubuntu-latest
75+
# Lint is advisory during the 0.8.x series — code passed tests on the
76+
# private fork before ruff rules were tightened. Will be promoted to
77+
# blocking once legacy lint debt is cleared.
78+
continue-on-error: true
79+
steps:
80+
- uses: actions/checkout@v4
81+
82+
- uses: actions/setup-python@v5
83+
with:
84+
python-version: "3.12"
85+
86+
- name: Install ruff
87+
run: python -m pip install ruff
88+
89+
- name: Ruff syntax check (pyflakes subset)
90+
run: ruff check --select E9,F63,F7,F82 .

‎CHANGELOG.md‎

Lines changed: 249 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,255 @@ All notable changes to **metal-guard** are documented here.
55
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
66
and this project adheres to [Semantic Versioning](https://semver.org/).
77

8+
## [0.9.0] — 2026-04-25
9+
10+
Minor release consolidating **panic #7–#11 findings** from Harper's
11+
production timeline (2026-04-16 → 2026-04-24) into the open-source
12+
distribution. Brings three new defences (B1 subprocess guard, C5
13+
cross-model cadence, C7 gemma-4 first-generate flush) and one new
14+
piece of advisory data: `KNOWN_PANIC_MODELS`.
15+
16+
**The honest caveat upfront.** metal-guard v0.9.0 narrows multiple race
17+
windows around the Apple IOGPU driver bug. It reduces panic frequency
18+
on every workload we've exercised. It **does not eliminate panic**
19+
on every model — specifically, `mlx-community/gemma-4-31b-it-8bit`
20+
still panicked on a production pipeline at Harper after every defence
21+
in this release was engaged (panic #11, 2026-04-24). When metal-guard
22+
is engaged and a model continues to panic in production, the right
23+
operational answer is to **switch backend** (Ollama / llama.cpp) or
24+
**pivot to a different model family** — see "When metal-guard is not
25+
enough" below.
26+
27+
### Added
28+
29+
- **`subprocess_inference_guard(model_id)` (B1).** Module-level
30+
contextmanager that wraps every `gen_fn(...)` call inside an MLX
31+
subprocess worker. Performs `mx.clear_cache()` PRE,
32+
`mx.synchronize()` POST, `mx.clear_cache()` POST, and emits
33+
`SUBPROC_PRE` / `SUBPROC_POST` breadcrumbs. Harper recorded 6
34+
consecutive subprocess-path kernel panics in 3 days (2026-04-20 →
35+
2026-04-23) before this guard; the streak ended on the first
36+
`gen_fn` invocation after it was wired in.
37+
38+
- **Cross-model cadence in `CadenceGuard` (C5).** Reject back-to-back
39+
loads of *different* models within a configurable window.
40+
`CadenceGuard(cross_model_interval_sec=…)` opts in; the default is
41+
`0.0` (disabled) to preserve v0.8.0 semantics. Env var
42+
`METALGUARD_CROSS_MODEL_INTERVAL` sets a process-wide default
43+
(resolver helper: `_resolve_cross_model_interval`). Violations raise
44+
`CrossModelCadenceViolation`, a subclass of `CadenceViolation` so
45+
existing `except CadenceViolation` keeps working.
46+
47+
- **Gemma-4 90-second floor (C5).** `mlx-community/gemma-4-*`,
48+
`unsloth/gemma-4-*`, and `mlx-models/gemma-4-*` always enforce a
49+
minimum 90 s cross-model cadence regardless of configured base.
50+
Constant: `GEMMA4_MIN_CROSS_MODEL_INTERVAL_SEC = 90.0`. All 8/8
51+
kernel panics in Harper's 2026-04 timeline with an identifiable
52+
at-panic model were in the gemma-4 family; panic #6 landed 66 s
53+
after prior unload, so 90 s = 66 s + ~36 % safety margin.
54+
55+
- **`gemma4_generation_flush(model_id, generate_call_count)` (C7).**
56+
First-generate settle window: `mx.synchronize()` +
57+
`mx.clear_cache()` + `time.sleep(3.0)` before the *first*
58+
`generate()` on a freshly-loaded gemma-4 worker. No-op on
59+
subsequent calls and on non-gemma-4 models. Env overrides:
60+
`METALGUARD_GEMMA4_FIRSTGEN_DISABLED=1`,
61+
`METALGUARD_GEMMA4_FIRSTGEN_SLEEP_SEC=<seconds>`. Harper's empirical
62+
breakdown: 7 of 8 gemma-4 panics landed on the first `generate()`
63+
within 7–66 s of worker-ready; the pre-existing flush barriers
64+
caught none of them.
65+
66+
**Renamed from the internal `gemma4_firstgen_guard`** — the name
67+
"guard" incorrectly suggested a block. This function is a flush +
68+
settle window, not a gate. If you need to gate, use
69+
`CircuitBreaker` or `require_cadence_clear`.
70+
71+
- **`KNOWN_PANIC_MODELS` advisory registry.** Module-level dict mapping
72+
model IDs to structured advisories (`panic_signature`,
73+
`reproductions`, `community`, `recommendation`, `upstream`).
74+
Companion helpers:
75+
- `check_known_panic_model(model_id) -> dict | None`
76+
- `warn_if_known_panic_model(model_id) -> bool` (idempotent; emits
77+
one `log.warning` per process per model)
78+
79+
Policy is the caller's — metal-guard does not refuse loads on its
80+
own. v0.9.0 ships with one entry: `mlx-community/gemma-4-31b-it-8bit`.
81+
82+
- **`require_cadence_clear(..., cross_model_interval_sec=...)`.** New
83+
keyword argument mirrors the `CadenceGuard` constructor param.
84+
`None` (default) delegates to `_resolve_cross_model_interval` which
85+
reads `METALGUARD_CROSS_MODEL_INTERVAL` or falls back to the C5
86+
default of 60 s. `0.0` disables. When `guard=` is supplied, the
87+
guard's own `cross_model_interval_sec` wins.
88+
89+
### Known affected models
90+
91+
#### `mlx-community/gemma-4-31b-it-8bit` — repeat offender
92+
93+
Two production kernel panics on Harper's box, 24 hours apart, same
94+
model, same pipeline, same panic signature:
95+
96+
| # | Date/time (local) | PID | Spawn → panic | Context |
97+
|-----|--------------------|-------|--------------:|-------------------------------------------|
98+
| 7 | 2026-04-23 03:14 | 67840 | ~6 min | rezivot pipeline, pre-C5 |
99+
| 11 | 2026-04-24 03:14 | 26608 | ~1.5 min | same pipeline as #7; post-C5 but ~1.5 min to panic anyway |
100+
101+
Signature for both: `IOGPUMemory.cpp:492 "completeMemory() prepare
102+
count underflow"`, Gen-4 hybrid attention, no concurrent generates.
103+
104+
**Community corroboration (all 2026-04):**
105+
106+
- [Hannecke — "MLX Crashed My Mac"](https://medium.com/@michael.hannecke/how-my-local-coding-agent-crashed-my-mac-and-what-i-learned-about-mlx-memory-management-e0cbad01553c)
107+
— M4 Max 64 GB, same panic signature, pivoted to
108+
`Qwen3-Coder-30B-A3B` MoE as workaround.
109+
- [`lmstudio-ai/lmstudio-bug-tracker#1740`](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1740)
110+
"Gemma-4 31b KV excessive KV cache footprint" — corroborates hybrid
111+
attention (50 sliding + 10 global) KV cache + 8-bit weights (~34 GB)
112+
+ full context KV (20 GB+) > 54 GB memory pressure on the 64 GB
113+
class. The thread documents 26 GB VRAM for a mere 8192 context.
114+
- [`ml-explore/mlx-lm#883`](https://github.com/ml-explore/mlx-lm/issues/883)
115+
— M3 Ultra 96 GB reports the same panic signature on the same model
116+
family.
117+
- [`ml-explore/mlx#3186` (comment 2026-04-24)](https://github.com/ml-explore/mlx/issues/3186#issuecomment-4314204974)
118+
— independent third-party data point: Mac mini M4 base 32 GB,
119+
macOS 26.4.1 (`25E253`), mlx 0.31.2, mlx-lm 0.31.3, model
120+
`mlx-community/Qwen3.6-35B-A3B-4bit`. Panic 8 min 16 s after
121+
`mlx_lm.server` start; `--prompt-cache-bytes 8 GiB` did not
122+
prevent it; reporter adopted `llama.cpp` for production serving.
123+
124+
**What this means in practice.** macOS 26.4.x has not fixed the bug.
125+
26.5 beta has not fixed the bug. `--prompt-cache-bytes` does not
126+
prevent it. Adding RAM to 96 GB does not prevent it. metal-guard
127+
v0.9.0 narrows the race windows but does not eliminate panic on this
128+
specific model in Harper's workload.
129+
130+
### When metal-guard is not enough
131+
132+
If you engage every defence in v0.9.0 (B1 + C5 + C7 + CircuitBreaker)
133+
and still observe repeat panics on the same model, that is a signal
134+
that the race window on that model is wider than metal-guard can
135+
narrow. Two escape hatches, in order of ROI:
136+
137+
1. **Switch backend.** Ollama and `llama.cpp` both use Metal MPS
138+
under the hood but run a persistent worker architecture that
139+
avoids the subprocess teardown race entirely. Harper's
140+
`harper-finance` project migrated to Ollama 2026-04-23 and has
141+
run zero-panic since. The independent `mlx#3186` M4-base
142+
reporter adopted `llama.cpp` for the same reason. You lose some
143+
raw throughput (MLX was measured 30–55 % faster on prefill in
144+
that report); you gain "doesn't panic the machine."
145+
146+
2. **Pivot to a different model family.** Mixture-of-Experts
147+
variants (e.g. `mlx-community/gemma-4-26b-a4b-it-4bit`,
148+
`Qwen3-Coder-30B-A3B`) have a much smaller active-parameter
149+
footprint per forward pass and a narrower KV growth trajectory.
150+
Community reports (Hannecke, lmstudio#1740) converge on MoE as
151+
the most reliable same-ecosystem workaround.
152+
153+
metal-guard is complementary to both — `subprocess_inference_guard`
154+
is useful even under Ollama if you spawn per-request subprocess
155+
workers, and `CadenceGuard` still helps regardless of backend when
156+
you hot-swap models.
157+
158+
### Panic timeline (Harper internal, 2026-04)
159+
160+
For calibration on what "engaged every defence" means in practice:
161+
162+
| # | Date (local) | Signature | Trigger | Defence landed |
163+
|-----|--------------|-------------------------------------------|----------------------------------------------------------|----------------------|
164+
| 7 | 2026-04-23 03:14 | `IOGPUMemory.cpp:492 prepare_count_underflow` | cross-model cadence not wired; gemma-4-31b-8bit | C5 phase-1 shipped |
165+
| 8 | 2026-04-23 14:07 | `IOGPUMemory.cpp:492` | GC / Metal async race across subprocess teardown | 0.8.6 hotfix 4-barrier |
166+
| 9 | 2026-04-23 17:40 | `IOGPUMemory.cpp:492` | Phase-2 +833-line regression | reverted via `git stash` |
167+
| 10 | 2026-04-23 19:46 | `IOGPUMemory.cpp:492` | interactive `python -c "import sentence_transformers"` for version verification → `torch` MPS backend init → process exit race → same Apple kernel bug | ad-hoc import SOP + `mlx-safe-python` wrapper |
168+
| 11 | 2026-04-24 03:14 | `IOGPUMemory.cpp:492` | same pipeline as #7; panic ~1.5 min after worker ready despite classic L9 in place | C7 flush + gemma-4 floor (this release) |
169+
170+
Panic #10 is worth calling out: it was triggered by a *verification*
171+
command on the host terminal, not by any MLX workload in production.
172+
Anything that imports `torch`, `mlx`, `mlx_lm`, `mlx_vlm`,
173+
`sentence_transformers`, `transformers`, `diffusers`, or
174+
`accelerate` initialises the Metal MPS backend and can walk into the
175+
same kernel bug at process exit. If your team uses metal-guard, the
176+
operational lesson is: during an active cooldown, verify package
177+
versions with `pip show <pkg>` or
178+
`python -c "import importlib.metadata as m; print(m.version('<pkg>'))"`
179+
— *never* `python -c "import <ml-pkg>; print(<ml-pkg>.__version__)"`.
180+
181+
### Fixed
182+
183+
- `prepare_count_underflow` panics on subprocess-isolated MLX workers
184+
(via B1 `subprocess_inference_guard`) — see v0.8.1 history above.
185+
- `prepare_count_underflow` panics on back-to-back loads of *different*
186+
models within seconds of each other — classic `CadenceViolation`
187+
only caught same-model patterns; `CrossModelCadenceViolation`
188+
extends coverage to the cross-model axis that Harper's panic #7
189+
exposed.
190+
- First-generate race window on gemma-4 family — `gemma4_generation_flush`
191+
inserts a mandatory synchronize + clear + sleep before the first
192+
forward pass, extending the settle window the four pre-existing
193+
flush barriers failed to cover.
194+
195+
### Changed
196+
197+
- `CadenceGuard.__init__` accepts a new keyword-only argument
198+
`cross_model_interval_sec` (default `0.0`, backwards-compat).
199+
Property `cadence_guard.cross_model_interval_sec` exposes the
200+
configured value.
201+
- `require_cadence_clear` accepts a new keyword-only argument
202+
`cross_model_interval_sec`. Behaviour for existing calls is
203+
unchanged unless the env var `METALGUARD_CROSS_MODEL_INTERVAL` is
204+
set.
205+
206+
### Migration
207+
208+
- **Pure additions on the public API surface.** Existing v0.8.0 code
209+
continues to work without changes.
210+
- **Env-var opt-in.** If you want cross-model cadence without code
211+
changes, set `METALGUARD_CROSS_MODEL_INTERVAL=60` in your
212+
environment and call `require_cadence_clear()` as before.
213+
- **Gemma-4 users get the 90-second floor automatically** the moment
214+
`cross_model_interval_sec > 0.0` (or the env var is set). This
215+
cannot be opted out of for the gemma-4 family by setting the base
216+
to 0 — the floor fires regardless. This is the one intentional
217+
asymmetry in the release and reflects the empirical panic data.
218+
219+
### Tests
220+
221+
- 38 new tests in `tests/test_v090_cross_model_cadence.py` covering
222+
`_is_gemma4_family` (13 parametrised cases),
223+
`_resolve_cross_model_interval` (6 cases: default / env / explicit
224+
/ invalid-fallback / negative-clamp / zero-disable),
225+
`CrossModelCadenceViolation` inheritance + fields,
226+
`CadenceGuard` cross-model check (same-model priority, zero-disabled
227+
fallthrough, zero-still-floors-gemma4, pass-after-interval),
228+
`require_cadence_clear` param plumbing,
229+
`gemma4_generation_flush` (non-gemma / count>0 / env-disabled /
230+
sleep-env-override / invalid-sleep-env-fallback),
231+
`KNOWN_PANIC_MODELS` hit/miss + `warn_if_known_panic_model`
232+
idempotence. Full suite: **204 passed** (166 pre-existing + 38
233+
new).
234+
235+
### Upstream references
236+
237+
Open upstream issues consistent with the v0.9.0 advisory content, all
238+
open at release time:
239+
240+
- [`ml-explore/mlx#3186`](https://github.com/ml-explore/mlx/issues/3186)
241+
— canonical subprocess isolation / `prepare_count_underflow` thread.
242+
Third-party corroboration (2026-04-24, M4 base 32 GB, Qwen3.6-35B-A3B):
243+
[`comment 4314204974`](https://github.com/ml-explore/mlx/issues/3186#issuecomment-4314204974).
244+
- [`ml-explore/mlx#3346`](https://github.com/ml-explore/mlx/issues/3346)
245+
— kernel panic reproducer catalogue.
246+
- [`ml-explore/mlx-lm#883`](https://github.com/ml-explore/mlx-lm/issues/883)
247+
— subprocess worker panic report (Hannecke, M4 Max 64 GB).
248+
- [`ml-explore/mlx-lm#1047`](https://github.com/ml-explore/mlx-lm/issues/1047)
249+
— Kimi K2.5 KV cache OOM on M3 Ultra.
250+
- [`ml-explore/mlx/#3267`](https://github.com/ml-explore/mlx/issues/3267)
251+
— GPU watchdog kills MLX when display is active (wontfix). Used to
252+
justify the `AGX_RELAX_CDM_CTXSTORE_TIMEOUT=1` import-time workaround
253+
kept from v0.2.x.
254+
255+
---
256+
8257
## [0.8.0] — 2026-04-17
9258

10259
Minor release porting **Layer 9 (L9)** from Harper's internal fork

0 commit comments

Comments
 (0)