feat(alerts): add alert for failed systemd services

[martinstenrose]

📃 Description

Adds a user-configurable Failed Services alert that notifies when any tracked systemd service enters the failed state, and again when all services recover. Resolves #1813.

The agent already reports per-service state, so no agent changes are needed – existing agents work unmodified.

Notes that may be worth a look during review:

• The SMART hook pattern doesn't apply. systemd_services rows are written via a raw batched SQL upsert, so PocketBase record hooks never fire for that collection. The check runs from HandleSystemAlerts instead, which is invoked on every update cycle. It's excluded from the numeric threshold loop, as Status already is.
• State is read from the systemd_services snapshot, not the update payload. Realtime dashboard subscriptions fetch from the agent with a shorter cache time, which the agent answers without systemd data – so an open dashboard overwrites the cached payload roughly once per second. Reading the payload makes the alert silently miss failures whenever someone is looking at the system page.
• Only rows from the most recent update count. The table is upserted and never pruned on change, so a unit that ceases to exist (a transient systemd-run unit after reset-failed) stops being reported but keeps its last known state until the retention sweep. Rows written in one cycle share a single updated timestamp, so the newest identifies what the agent last reported. The services table applies the same rule – it renders only rows within 70s of the newest – so this keeps the alert and the table in agreement.
• The alert fires on first observation rather than offering a "for N minutes" delay. The agent refreshes systemd state every 10 minutes, so a shorter delay would expire against data identical to what scheduled it and could never cancel – latency with no filtering. A failed unit is also a sticky state rather than a flapping one, so there is less to debounce than for Status. The duration slider is hidden via a new noDuration flag rather than presenting a control that cannot affect the outcome.
• Open question: if the agent's systemd interval ever becomes shorter or configurable, a delay would be worth having. Happy to restore the slider if you'd prefer it – the min field and slider code are untouched.

Granularity is system-wide ("any tracked service failed"), consistent with the existing Disk/Temperature/GPU alerts, with failed service names listed in the notification body. SERVICE_PATTERNS on the agent already scopes what is tracked.

The second commit, fix(hub): remove systemd services that are no longer reported, fixes a pre-existing data-correctness bug that this feature surfaced - see the Fixed section. It's independent of the alert; happy to split it into its own PR if you'd prefer, the alert is correct either way.

Locale catalogs are left untouched, since translations are handled via Crowdin – the two new strings are t macros in lib/alerts.ts and will be picked up by the next extraction.

Verified against two hosts: single and multiple failures, recovery, and correct alerting while a stale orphaned row was present. The row removal in the second commit is covered by unit tests rather than live testing.

Note: go test ./internal/alerts/ currently fails on a pre-existing teardown race unrelated to this change – a StartUpdater goroutine sleeps 11s (system.go:82) then calls setDown() against a hub whose test DB is already closed. The existing suite finishes at ~13.1s, right as those timers fire, so appending any test exposes it. Reproducible on a clean checkout with go test -tags testing ./internal/alerts/ -count=2. The new tests pass in isolation (-runSystemd). Happy to open a separate issue for it.

Note: Written with the help of an AI agent (Claude Code). The design decisions, review, and testing against my own hosts are mine; the commits carry Co-Authored-By trailers.

📖 Documentation

henrygd/beszel-docs#64

🪵 Changelog

➕ Added

• New Failed Services (SystemdFailed) alert type – triggers when any tracked systemd service enters the failed state, listing the failed service names in the notification
• Recovery notification when all previously failed services return to a non-failed state
• Migration adding SystemdFailed to the alerts.name select field
• noDuration flag on AlertInfo, so alert types that fire on first observation don't render a duration slider
• Startup pass that clears stale triggered systemd alerts after a hub restart
• Test coverage in internal/alerts/alerts_systemd_test.go (12 cases: immediate fire, full fail/recover cycle, re-notification suppression, repeated-failure idempotence, orphaned rows, missing-data handling, multi-user, stale-state resolution)

✏️ Changed

• HandleSystemAlerts now dispatches systemd alerts and excludes SystemdFailed from the numeric threshold loop
• Alert config sheet hides the threshold and duration controls when an alert type configures neither, instead of rendering an empty section
• min defaults to 0 for alert types that fire immediately, rather than storing an unused 10 minute delay

🔧 Fixed

• A service removed from a host could keep showing as failed for over an hour, and the two views disagreed about it in the meantime. Nothing removed rows for services the agent stopped reporting, so systemd_services kept the last known state until the retention sweep. The all-systems dashboard counts services from the agent and dropped it immediately, while the per-system services table reads the stored rows and kept showing it – e.g. a dashboard reading 44 (failed: 0)against a services tab showing Total: 45 / Failed: 1 for the same host. Rows are now deleted at write time once the agent stops reporting them.

📷 Screenshots

[apizz]

+1 for per-service failure selection please 🙂

[martinstenrose]

+1 for per-service failure selection please 🙂

Thanks for the feedback! I went with system-wide mainly because it was much simpler to implement and covers what I actually needed. I think it’s also the more common case – you generally don’t want any service left in a failed state, and reset-failed is always available if you want to clear one you don’t care about.

SERVICE_PATTERNS on the agent can already be used to limit tracking to a specific set of services, which gets you part of the way there.

Given limited time, I opted for the simpler system-wide approach for now – per-service selection could be added as a follow-up if there’s demand.

[apizz]

Yep, just wanted to indicate here since in the fr issue you said to 🙂

[martinstenrose]

Changed the icon from ActivityIcon to ServerCrashIcon.