Skip to content

Commit 9266347

Browse files
committed
fix(camera): distinguish an unreachable on-demand camera from a sleeping one
An on-demand camera delivers no frames whether it is resting between events or refusing the connection, so deriving `sleeping` from frame arrival alone reported every fault on a battery camera as a healthy nap. Nothing else contradicted it: the retry loop logged failures at debug level, which the process never emits, and reconnect counters exclude on-demand sources by design. The RTSP source now files the outcome of every attempt and separates a server answering "no stream at this path" from one refusing, rejecting, or ignoring us. A camera failing for the latter reason reports sleeping=false with the redacted reason in stream_error, and its source logs a rate-limited warning rather than a debug line. Per-camera last-connect time is exposed on the camera detail and as vedetta_camera_last_connected_timestamp_seconds. That is the only signal separating a camera napping between events from one that has been unplugged or dropped from its bridge, since both report the stream as unpublished indefinitely. It is absent rather than zero for a camera that has never connected, so its age cannot be read as decades of uptime.
1 parent dbd80ec commit 9266347

10 files changed

Lines changed: 730 additions & 175 deletions

File tree

internal/api/generated.go

Lines changed: 183 additions & 170 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

internal/api/handler_cameras.go

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ func (s *Server) GetCamera(w http.ResponseWriter, r *http.Request, name string)
6464
st := cam.Status()
6565
_, hasPTZ := s.ptzClients[name]
6666
zones, _ := s.db.ListZones(name)
67-
writeJSON(w, http.StatusOK, map[string]any{
67+
resp := map[string]any{
6868
"name": st.Name,
6969
"online": st.Online,
7070
"sleeping": st.Sleeping,
@@ -76,7 +76,17 @@ func (s *Server) GetCamera(w http.ResponseWriter, r *http.Request, name string)
7676
"zone_count": len(zones),
7777
"recording": s.recorder != nil,
7878
"source_fps": st.SourceFPS,
79-
})
79+
}
80+
// Both keys are absent rather than empty when there is nothing to report: a
81+
// camera that has never connected has no last-connected time, and a zero
82+
// timestamp there would render as an age of decades.
83+
if !st.LastConnected.IsZero() {
84+
resp["last_connected"] = st.LastConnected
85+
}
86+
if st.StreamError != "" {
87+
resp["stream_error"] = st.StreamError
88+
}
89+
writeJSON(w, http.StatusOK, resp)
8090
}
8191

8292
func (s *Server) SendPTZCommand(w http.ResponseWriter, r *http.Request, name string) {

internal/api/handler_health.go

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,24 @@ func (s *Server) GetMetrics(w http.ResponseWriter, _ *http.Request) {
274274
}
275275
}
276276

277+
// The only signal that separates an on-demand camera napping between events
278+
// from one that has been unplugged or dropped from its bridge: both report
279+
// online=0 sleeping=1 indefinitely, and nothing else ever changes. Alert on
280+
// time() minus this exceeding the longest plausible gap between events.
281+
//
282+
// A camera that has not connected since startup has no timestamp to report,
283+
// so its sample is omitted rather than written as 0, which would assert a
284+
// successful connection in 1970 and make the alert above read as healthy.
285+
// Detect that case with absent(), against vedetta_camera_online, which is
286+
// emitted for every camera.
287+
fmt.Fprintf(&b, "# HELP vedetta_camera_last_connected_timestamp_seconds Unix time of the camera's last successful RTSP connection. Absent when the camera has not connected since startup.\n# TYPE vedetta_camera_last_connected_timestamp_seconds gauge\n")
288+
for _, st := range cameraStatuses {
289+
if st.LastConnected.IsZero() {
290+
continue
291+
}
292+
fmt.Fprintf(&b, "vedetta_camera_last_connected_timestamp_seconds{camera=%q} %d\n", promLabel(st.Name), st.LastConnected.Unix())
293+
}
294+
277295
// Drop-on-full fan-out counters: the detection-overlay SSE hub and the MSE
278296
// pipeline shed frames to slow clients rather than blocking. A rising count
279297
// means live overlay / playback is silently degrading for those viewers.

internal/api/openapi.yaml

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2779,7 +2779,11 @@ components:
27792779
also reports online=false, because no frames are arriving, but it is
27802780
not a fault. A camera with online=false and sleeping=false is a
27812781
genuine outage. Always false for a mains-powered camera, and always
2782-
false while the camera is stopped.
2782+
false while the camera is stopped. Also false when the camera's RTSP
2783+
source is failing for a reason other than its stream not being
2784+
published (refused connection, rejected credentials, timeout): that
2785+
delivers no frames just as sleep does, but it is an outage, and
2786+
stream_error on the camera detail carries the reason.
27832787
stopped:
27842788
type: boolean
27852789
description: >-
@@ -2844,6 +2848,24 @@ components:
28442848
type: number
28452849
format: float
28462850
description: Rolling-window source frame rate measured at the detect consumer (frames per second).
2851+
last_connected:
2852+
type: string
2853+
format: date-time
2854+
description: >-
2855+
Time of the camera's last successful RTSP connection. Absent when it
2856+
has not connected since Vedetta started; absent is not the same as
2857+
old, and must not be rendered as an age. For an on-demand camera
2858+
this is the only signal separating one napping between events from
2859+
one that has been unplugged or removed from its bridge, since both
2860+
report the stream as unpublished indefinitely.
2861+
stream_error:
2862+
type: string
2863+
description: >-
2864+
Reason the camera's RTSP source cannot connect, with credentials
2865+
redacted. Present only when the source has failed repeatedly for a
2866+
reason other than its stream not being published, so it is absent
2867+
both while the camera is connected and while an on-demand camera is
2868+
resting between events.
28472869
28482870
Zone:
28492871
type: object

internal/camera/camera.go

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,20 @@ type CameraStatus struct {
211211
// worth alerting on. Always false for a normal mains-powered camera, whose
212212
// stream really should be there.
213213
Sleeping bool `json:"sleeping"`
214+
// LastConnected dates the last time this camera's RTSP source reached PLAY.
215+
// Omitted when it has not connected since Vedetta started, which is a
216+
// different fact from a zero time and must not be rendered as an age.
217+
//
218+
// For an on-demand camera this is the only signal separating one napping
219+
// between events from one that has been unplugged or removed from its
220+
// bridge: both answer "no such stream" indefinitely, so a stale timestamp
221+
// here is the whole warning.
222+
LastConnected time.Time `json:"last_connected,omitempty"`
223+
// StreamError carries the most recent RTSP connection failure when it is
224+
// something other than the stream not being published, with credentials
225+
// redacted. Empty while connected, and empty for an on-demand camera
226+
// resting between events, because that is not a fault.
227+
StreamError string `json:"stream_error,omitempty"`
214228
}
215229

216230
func NewCamera(cfg config.CameraConfig, detector *detect.Detector, motion config.MotionConfig, events chan<- Event, eventEnds chan<- EventEnd, presenceEvents chan<- PresenceEvent, hub *rtsp.Hub, snapshotPath string, snapshotQuality int, recordingPath string, faceRecognizer *detect.FaceRecognizer, faceEvents chan<- FaceEvent, faceCropDir string, motionActivity chan<- MotionActivity, detections chan<- DetectionFrame) *Camera {
@@ -1050,6 +1064,25 @@ func (c *Camera) Status() CameraStatus {
10501064
if lastSeen.IsZero() {
10511065
lastSeen = c.cachedSnapshotTime
10521066
}
1067+
1068+
// An on-demand camera delivering no frames is assumed asleep, but that
1069+
// assumption is only sound while the failures are the stream not being
1070+
// published. One that is refusing the connection or rejecting credentials
1071+
// delivers no frames either, and reporting it as a nap hides the fault
1072+
// behind a badge that says everything is fine.
1073+
sleeping := c.config.OnDemand && !online
1074+
var lastConnected time.Time
1075+
var streamErr string
1076+
if c.hub != nil {
1077+
if h, ok := c.hub.Health(c.config.URL); ok {
1078+
lastConnected = h.LastConnected
1079+
if h.Faulted() {
1080+
streamErr = h.LastError
1081+
sleeping = false
1082+
}
1083+
}
1084+
}
1085+
10531086
return CameraStatus{
10541087
Name: c.config.Name,
10551088
Online: online,
@@ -1059,7 +1092,9 @@ func (c *Camera) Status() CameraStatus {
10591092
Degraded: c.degradedReason != "",
10601093
DegradedReason: c.degradedReason,
10611094
SourceFPS: fps,
1062-
Sleeping: c.config.OnDemand && !online,
1095+
Sleeping: sleeping,
1096+
LastConnected: lastConnected,
1097+
StreamError: streamErr,
10631098
}
10641099
}
10651100

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
package camera
2+
3+
import (
4+
"context"
5+
"testing"
6+
7+
"github.com/bluenviron/gortsplib/v5/pkg/base"
8+
"github.com/bluenviron/gortsplib/v5/pkg/liberrors"
9+
10+
"github.com/rvben/vedetta/internal/rtsp"
11+
)
12+
13+
const testSourceURL = "rtsp://198.51.100.20:554/live0"
14+
15+
// seedOnDemandSource marks cam as on-demand and wires it to a hub holding a
16+
// Source for testSourceURL that has already recorded n failed attempts with
17+
// err. Nothing is dialled.
18+
func seedOnDemandSource(t *testing.T, cam *Camera, n int, err error) {
19+
t.Helper()
20+
hub := rtsp.NewHub(context.Background())
21+
t.Cleanup(hub.Close)
22+
src := rtsp.NewSource(testSourceURL)
23+
hub.SetSourceForTest(testSourceURL, src)
24+
for range n {
25+
src.SimulateAttemptForTest(err)
26+
}
27+
cam.mu.Lock()
28+
cam.hub = hub
29+
cam.config.URL = testSourceURL
30+
cam.config.OnDemand = true
31+
cam.mu.Unlock()
32+
}
33+
34+
// An on-demand camera reports no frames whether it is asleep or misconfigured,
35+
// so Sleeping cannot be derived from frame arrival alone. Deriving it that way
36+
// is what let a wrong password on a battery camera sit behind a badge saying
37+
// the camera was merely resting, with nothing in the logs or metrics to
38+
// contradict it.
39+
func TestStatusOnDemandFaultIsNotSleeping(t *testing.T) {
40+
asleep := NewTestCamera("battery")
41+
seedOnDemandSource(t, asleep, 5, liberrors.ErrClientBadStatusCode{
42+
Code: base.StatusNotFound, Message: "Stream Not Found",
43+
})
44+
st := asleep.Status()
45+
if !st.Sleeping {
46+
t.Errorf("sleeping = false for an on-demand camera whose stream is unpublished, want true: that is its resting state")
47+
}
48+
if st.StreamError != "" {
49+
t.Errorf("stream_error = %q for a resting camera, want empty: an operator must not be sent after a fault that does not exist", st.StreamError)
50+
}
51+
52+
broken := NewTestCamera("battery-broken")
53+
seedOnDemandSource(t, broken, 5, liberrors.ErrClientBadStatusCode{
54+
Code: base.StatusUnauthorized, Message: "Unauthorized",
55+
})
56+
st = broken.Status()
57+
if st.Sleeping {
58+
t.Errorf("sleeping = true for an on-demand camera whose credentials are rejected, want false: this is an outage reported as a nap")
59+
}
60+
if st.Online {
61+
t.Errorf("online = true for a camera that never connected, want false")
62+
}
63+
if st.StreamError == "" {
64+
t.Errorf("stream_error empty for a rejected connection, want the reason the camera cannot be reached")
65+
}
66+
}
67+
68+
// A camera that has never connected has no last-connected time. Reporting the
69+
// zero value would give consumers a timestamp in year one, whose age reads as a
70+
// plausible-looking two millennia rather than as missing data.
71+
func TestStatusLastConnectedAbsentUntilFirstConnection(t *testing.T) {
72+
cam := NewTestCamera("battery")
73+
seedOnDemandSource(t, cam, 1, liberrors.ErrClientBadStatusCode{
74+
Code: base.StatusNotFound, Message: "Stream Not Found",
75+
})
76+
if got := cam.Status().LastConnected; !got.IsZero() {
77+
t.Errorf("last_connected = %v for a camera that has never connected, want the zero value so the field is omitted", got)
78+
}
79+
}
80+
81+
// A camera with no Source yet (nothing has opened the stream) must not be
82+
// reported as faulted. Nothing has tried, which is a different fact from
83+
// having tried and failed.
84+
func TestStatusNoSourceIsNotFaulted(t *testing.T) {
85+
cam := NewTestCamera("battery")
86+
cam.mu.Lock()
87+
cam.config.OnDemand = true
88+
cam.mu.Unlock()
89+
90+
st := cam.Status()
91+
if st.StreamError != "" {
92+
t.Errorf("stream_error = %q with no source created yet, want empty", st.StreamError)
93+
}
94+
if !st.Sleeping {
95+
t.Errorf("sleeping = false for an on-demand camera with no source yet, want true")
96+
}
97+
}

internal/rtsp/health.go

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
package rtsp
2+
3+
import (
4+
"errors"
5+
"net/url"
6+
"strings"
7+
"time"
8+
9+
"github.com/bluenviron/gortsplib/v5/pkg/base"
10+
"github.com/bluenviron/gortsplib/v5/pkg/liberrors"
11+
)
12+
13+
// SourceHealth is a point-in-time view of a Source's connection attempts.
14+
//
15+
// Frame arrival alone cannot tell an on-demand camera resting between events
16+
// from one that is awake and rejecting us: both deliver nothing. The difference
17+
// is in how the connection attempt fails, which only the Source sees, so it has
18+
// to be recorded here and carried out to the API and the logs.
19+
type SourceHealth struct {
20+
Connected bool
21+
// LastConnected is when the source last reached PLAY. Zero means it has not
22+
// connected since this Source was created. A camera whose stream is simply
23+
// never published (unplugged, removed from its bridge, wrong path) looks
24+
// exactly like a sleeping one on every other signal; the age of this
25+
// timestamp is what separates them.
26+
LastConnected time.Time
27+
LastAttempt time.Time
28+
// LastError is the failure from the most recent attempt that did not reach
29+
// PLAY, with any credentials from the URL redacted. Empty after a success.
30+
LastError string
31+
// Unpublished records that LastError was the server answering "no stream at
32+
// this path" rather than refusing, rejecting, or ignoring us.
33+
Unpublished bool
34+
// ConsecutiveFailures counts attempts since the last successful connection.
35+
// An attempt that reached PLAY and then dropped resets it: that is a
36+
// disconnection, not a failure to reach the camera.
37+
ConsecutiveFailures int64
38+
}
39+
40+
// onDemandFaultThreshold is how many consecutive non-routine failures a source
41+
// must accumulate before Faulted reports it as broken. At onDemandRetry that is
42+
// roughly ten seconds: long enough to ride out a bridge reboot without flapping
43+
// a camera between SLEEPING and OFFLINE, short enough that a wrong password
44+
// surfaces while the operator is still looking at the screen.
45+
const onDemandFaultThreshold = 3
46+
47+
// Faulted reports whether the source is failing for a reason other than its
48+
// stream not being published.
49+
//
50+
// This is the distinction the rest of the system needs. A battery camera behind
51+
// a bridge answers 404 between events and that is its resting state; the same
52+
// camera answering 401, refusing the connection, or timing out is broken. Both
53+
// deliver zero frames, so without this every misconfiguration on an on-demand
54+
// camera reads as a healthy nap and nothing ever reports it.
55+
func (h SourceHealth) Faulted() bool {
56+
return !h.Connected &&
57+
!h.Unpublished &&
58+
h.LastError != "" &&
59+
h.ConsecutiveFailures >= onDemandFaultThreshold
60+
}
61+
62+
// streamUnpublished reports whether err is the server saying it has no stream
63+
// at this path. Anything else (a refused connection, rejected credentials, a
64+
// timeout) means the far end is reachable-but-wrong or not reachable at all,
65+
// which is a fault however the camera is powered.
66+
func streamUnpublished(err error) bool {
67+
var bad liberrors.ErrClientBadStatusCode
68+
if !errors.As(err, &bad) {
69+
return false
70+
}
71+
return bad.Code == base.StatusNotFound
72+
}
73+
74+
// redactCredentials removes RTSP userinfo from a message destined for a log
75+
// line or the HTTP API. Error strings from the client library quote the URL
76+
// they were given, which carries the camera password.
77+
//
78+
// Only whole credential-bearing substrings are replaced. Substituting the
79+
// username or password wherever they appear would mangle unrelated text when
80+
// either is a short or common word.
81+
func redactCredentials(msg, rawURL string) string {
82+
if msg == "" || rawURL == "" {
83+
return msg
84+
}
85+
msg = strings.ReplaceAll(msg, rawURL, SanitizeURL(rawURL))
86+
u, err := url.Parse(rawURL)
87+
if err != nil || u.User == nil {
88+
return msg
89+
}
90+
msg = strings.ReplaceAll(msg, u.User.String()+"@", "***:***@")
91+
// url.UserPassword percent-encodes on String(); the URL may have been
92+
// configured with the literal form, which is what the library echoes back.
93+
if pw, ok := u.User.Password(); ok {
94+
msg = strings.ReplaceAll(msg, u.User.Username()+":"+pw+"@", "***:***@")
95+
}
96+
return msg
97+
}

0 commit comments

Comments
 (0)