EchOnyx 1.0 Readiness — Audit, Support Matrix, and Checklist

Phase 0 audit performed 2026-07-06 on the Strix Halo reference machine (Ryzen AI MAX+ 395, 128 GB, ROCm). This document is the living tracker for the 1.0 push: findings are checked off as they land.

How this audit was performed

Read every doc (README.md, backend/README.md, AGENTS.md, project_status_and_backlog.md), all compose files, scripts, the full frontend, and the backend config/route surface.
Ran the canonical backend test command, frontend lint + production build.
Booted the backend against disposable Postgres/Redis containers with a CPU profile and walked the real first-run flow over HTTP: auth setup, login/session, settings/hardware/model-status, valid + invalid upload, job listing, delete.
Not exercised live in this audit: full model pipeline on ROCm (requires multi-GB model downloads; validated separately per the acceptance log), NVIDIA and Apple paths (no such hardware attached — status taken from the repo's own live-validation log and marked accordingly below).

Verification status at audit time

Check	Result
`uv run pytest backend/tests` (canonical command)	FAILS collection on the reference machine (see F1)
`HARDWARE_PROFILE=cpu_only GPU_BACKEND=cpu uv run pytest backend/tests`	197 passed, 2 failed (F2, F3)
`npm run lint` (frontend)	clean
`npm run build` (frontend)	clean
First-run auth → upload → jobs → delete via HTTP	works
Invalid upload rejection (non-video file)	works

Prioritized findings

P0 — broken at HEAD / reproducibility blockers

F1. Test suite cannot run on the reference machine. get_settings() probes real hardware at import time and validate_hardware_requirements raises when Strix Halo is detected without ROCm torch (backend/app/config.py:599, :214-225), so uv run pytest backend/tests dies with 15 collection errors on the very machine AGENTS.md names as the review surface. Tests must pin a hardware profile (conftest env override) so the suite is deterministic everywhere.
F2. Author-machine path baked into a test. backend/tests/test_config.py:283 asserted a developer-specific upload path; fixed to assert relative to the detected project root.
F3. HEAD committed with a red test. backend/tests/test_nvidia_runtime_contract.py:10 still expects the vLLM v0.11.2 image; docker-compose.nvidia.yml was bumped to v0.17.1 in c82d8a1 without updating the contract test.
F4. Dead script reference. scripts/setup.sh no longer references the removed ./scripts/download-models.sh; setup now points users to Settings → Model Downloads or first-processing auto-downloads when MODEL_AUTO_DOWNLOAD=true, and prints health/readiness self-checks.
F5. Dependency locking added. frontend/package-lock.json and backend/uv.lock are present, Dockerfiles install from those locks (npm ci / frozen uv export), llama-cpp-python is pinned to v0.3.33, and the external llama.cpp build default is pinned to b9888. Optional NeMo git installation remains intentionally unchanged.
F6. No CI. GitHub Actions now runs backend tests, a scoped backend ruff gate, frontend lint/typecheck/build, frontend Docker build smoke, and compose config validation on push and pull requests to main.
F7. .gitignore misses node_modules/ and .next/, so a local frontend install dirties the tree.

P1 — product completeness / UI–backend drift

P2 — docs, privacy, ops polish

F23. Private identifiers throughout. Internal IPs, hostnames, and user paths across README.md, backend/README.md, AGENTS.md, project_status_and_backlog.md, and two test files were replaced with documentation placeholders or generic test fixtures without gutting the examples.
F24. Docs are an operator changelog, not user docs. README is now a newcomer-facing user guide, dated validation history moved to CHANGELOG.md, future work moved to ROADMAP.md, CONTRIBUTING.md and docs/ARCHITECTURE.md were added, and project_status_and_backlog.md now points to the canonical docs.
F25. Config drift. .env.example and compose defaults now match the registry-backed Qwen3-30B-A3B-Q4_K_M.gguf filename and the auto-attached mmproj-Qwen3VL-32B-Instruct-Q8_0.gguf; docker-compose.yml no longer uses the deprecated top-level version: key.
F26. Real readiness probe. /ready reports database, Redis, Chroma, and Celery worker heartbeat status. Database/Redis/Chroma failures return 503; a missing worker heartbeat returns 200 with status: degraded because the API can still serve read-only traffic.
F27. AMD GPU detection reports useful memory. ROCm CSV parsing now reads VRAM and GTT used/total pools, sets AMD available memory, and prefers Linux system memory for unified-memory APUs while keeping the Strix Halo unified_memory_gb >= 96 profile heuristic unchanged.
F28. Compose hardcodes host-specific group ids. docker-compose.amd.yml uses ${VIDEO_GROUP_ID:-44} and ${RENDER_GROUP_ID:-993} defaults, with .env.example guidance for discovering host ids via getent group video render | cut -d: -f3.

Support matrix (proposed → adopted)

Path	Tier	Rationale / caveats
AMD Strix Halo (ROCm 7.2, Docker: base + `docker-compose.amd.yml`)	Tier 1	Reference platform; live-validated repeatedly; ROCm-only contract enforced in code. Must pass acceptance before 1.0.
NVIDIA single/multi-GPU (CUDA, Docker: base + `docker-compose.nvidia.yml`)	Tier 1	Live-validated on a mixed 3090 + RTX PRO 6000 host (March 2026 acceptance log). No NVIDIA hardware in this audit environment; CI covers image build + contract tests, not GPU runtime.
Apple Silicon (Metal, host-run, small-model defaults)	Tier 2	Validated on a 16 GB Mac mini per repo log; host-run only (no Metal in Docker), Celery `--pool=solo` required. Documented caveats.
CPU-only (Docker base compose or host)	Tier 2	Pipeline runs but the default large-model set is impractical; supported for small models/short clips and functional testing. Needs small-profile defaults (F20).
External OpenAI-compatible endpoints (vision/summarization offload)	Tier 2	Explicitly opt-in, keeps local-first default; validated against llama.cpp/vLLM servers.
ROCm vLLM runtime (`ROCM_LLM_RUNTIME=vllm`, `INSTALL_VLLM=1`)	Experimental	Repo's own findings: fragile on gfx1151, slow startup, teardown hazards. Keep available, label clearly.
Canary / Granite ASR (`INSTALL_NEMO=1`, granite models)	Experimental	Alternative ASR families; not part of the default path.
Vulkan backend enum	Experimental/deprecated	Rejected on Strix Halo by contract; kept only as an env value. Document as unsupported.

1.0 readiness checklist

Launch-blocking

Suite green everywhere: fix F1–F3; one canonical test command documented and used in CI.
Reproducible builds: package-lock.json + uv.lock committed; Docker builds install from locks; llama-cpp-python pinned to a rev; moving model/runtime deps pinned or documented (F5).
CI on every push: backend pytest, frontend lint + typecheck + build, Docker build smoke, plus basic frontend tests for critical flows (F6).
UI redesign to a coherent design system with dialogs/toasts, loading/empty/error states, light/dark, responsive layout, baseline a11y (F13–F19, F15).
Close UI↔backend drift: batch UI, job cancel, similar videos, slide images, duplicate visibility (F8–F12, F21).
Guided first-run: hardware-aware recommended model sets, opt-in downloads with progress/disk/VRAM checks, HF token + pyannote license guidance, fixed setup script (F4, F20).
Docs split: newcomer README with per-platform quickstarts, CHANGELOG, ROADMAP, CONTRIBUTING, architecture overview; privacy scrub with placeholders (F23–F25).
Ops: /ready endpoint with dependency checks; every setting documented + validated; scripts/acceptance.sh kept canonical (F26).
Remove the WIP banner (F22) once the above hold.

Post-1.0 / explicitly not blocking

CLAP fixture promotion for meeting/software-demo separation (research).
Dynamic residency planner improvements beyond current free-memory placement.
Incremental recomputation / step-level caching; multi-index retrieval routing; live transcription; action-item external sync.
Per-user auth/RBAC, MFA, session-device management (single-admin model is intentional at 1.0).
ROCm vLLM as a default runtime.

Decisions taken (surfaced, with rationale)

Job progress stays on polling for 1.0 (F11): the WS route remains for automation, but the redesigned UI keeps interval polling — one mechanism to test, no reconnect/auth complexity before launch.
CPU-only is Tier 2, not Tier 1: the default model set cannot meet a "first successful result" bar on CPU; rather than shrink the product, we ship small-profile defaults + honest docs.
NVIDIA stays Tier 1 with a documented verification boundary: no NVIDIA hardware is attached to this audit/CI environment; runtime claims rest on the March 2026 live acceptance record, guarded by contract tests and image build CI.

Final status (2026-07-06)

All findings F1–F28 and every launch-blocking checklist item are closed; CI is green on main. The full 1.0 readiness report (what was tested, what was not, and remaining risks) is summarized here:

Verified in this cycle: backend suite (224 tests) on the Strix Halo reference machine; frontend lint/typecheck/build plus 17 component tests; live HTTP walk of auth bootstrap, upload (valid + rejected), batch create/list, labels, recommendations, download validation, tags, delete, /health, and /ready; all frontend routes render against a live backend; Docker compose configs validate and the frontend image builds in CI.
Not verified here (documented boundary): GPU model inference end-to-end on ROCm/CUDA/Metal — no idle GPU/Apple hardware was available to this cycle. These paths keep their contract tests, pinned runtimes, and prior live-acceptance record; rerun scripts/acceptance.sh on live targets before tagging a release.
Remaining risks: the ruff CI gate covers only fatal/syntax rules (~400 legacy style violations to burn down); ROCm vLLM stays experimental; the packaged CLAP calibration remains conservative; single-admin auth is an intentional 1.0 constraint (RBAC/MFA on the roadmap).

Post-review fixes (2026-07-06)

Slide image serving is contained to the per-video pipeline frames directory and rejects poisoned database paths or symlink escapes.
GGUF model resolution accepts bare filenames, case-insensitive existing files, and clear read-only download-target failures.
Video mutation responses preserve sanitized duplicate metadata.
Batch creation reports accepted videos plus per-file rejection reasons.
/ready?strict=1 returns 503 for any failing dependency, including the worker heartbeat.
Setup hardware detection now classifies high-RAM ROCm systems as strix_halo before falling back to generic multi_gpu.
Live Strix Docker validation: first-run admin bootstrap now uses scripts/bootstrap-admin.sh and the setup rejection points operators to supported bootstrap paths.
Live NVIDIA ai-server validation: managed endpoint GPU pins keep host indices in CUDA_VISIBLE_DEVICES while passing CUDA-local ordinals to llama.cpp and vLLM.
Live NVIDIA re-tests (two rounds): CUDA_VISIBLE_DEVICES does not compose across processes — a child re-enumerates the container's physical device namespace — so GPU pins are exported verbatim as physical ids with engine arguments translated to ordinals within the child's own list; a parent CUDA_VISIBLE_DEVICES acts as a deployment allowlist and unresolvable pins fail fast instead of silently falling back to device 0. Vision no longer collides with the summarization card.
Model status now uses the shared local GGUF resolver, including bare-name and case-insensitive cache hits.
The standalone llama.cpp server wrapper translates extra GPU selector args after role pinning narrows CUDA_VISIBLE_DEVICES.
Live NVIDIA end-to-end validation: worker-side planner GPU host indices now translate to CUDA-local ordinals under narrowed CUDA_VISIBLE_DEVICES.
All-invalid batch uploads now return the accumulated per-file rejection reasons in the 400 detail.

Live NVIDIA end-to-end result (2026-07-06, f3303a4)

Full single-video acceptance PASSED on the multi-GPU NVIDIA host, 3090s only: transcription/embeddings on one card, summarization (Qwen3-30B Q4_K_M at 16k context) on a second, vision (Qwen2.5-VL-7B via vLLM) on a third, with the RTX PRO 6000s and co-tenant workloads untouched throughout and clean teardown. This closes the GPU verification boundary noted above for the NVIDIA Tier 1 path (ROCm inference verification still pends a free window on the Strix reference box). Post-1.0 refinements noted from the run: the runtime planner could pre-filter candidate GPUs to the process's CUDA_VISIBLE_DEVICES so the out-of-visibility fallback (first visible device) is never needed, and first search/ask requests can return one transient 500 before warmup — both cosmetic, neither blocking.

FilesExpand file tree

1.0-READINESS.md

Latest commit

History