From a83c066e0dea5fe73ddccb173d15f37e858ec080 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Thu, 12 Mar 2026 20:01:43 +0000
Subject: [PATCH 1/2] Initial plan


From 8d4d7e4694ce5c5ba662126b22686e63978f546d Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Thu, 12 Mar 2026 20:11:26 +0000
Subject: [PATCH 2/2] Update rosbag_blackbox README.md to match latest
 implementation

Co-authored-by: marc-hanheide <1153084+marc-hanheide@users.noreply.github.com>
---
 src/rosbag_blackbox/README.md | 298 +++++++++++++++++++++++++++-------
 1 file changed, 236 insertions(+), 62 deletions(-)

diff --git a/src/rosbag_blackbox/README.md b/src/rosbag_blackbox/README.md
index 65253d1..234fedb 100644
--- a/src/rosbag_blackbox/README.md
+++ b/src/rosbag_blackbox/README.md
@@ -3,7 +3,8 @@
 A ROS 2 **Humble** package that provides:
 
 1. **Ring Buffer Node** – continuously records configured topics into an
-   in-memory ring buffer and saves a *snapshot* (rosbag file) on demand.
+   in-memory ring buffer and saves a *snapshot* (rosbag file) on demand or
+   automatically on a sliding-window schedule.
 2. **Snapshot Server** – a combined **FastAPI** REST + **FastMCP** (Model
    Context Protocol) server that lets REST clients and AI agents access and
    introspect snapshot files.
@@ -16,6 +17,11 @@ A ROS 2 **Humble** package that provides:
 - [Installation](#installation)
 - [Quick Start](#quick-start)
 - [Configuration Reference](#configuration-reference)
+  - [Buffer Settings](#buffer-settings)
+  - [Snapshot Output](#snapshot-output)
+  - [Automatic Snapshots](#automatic-snapshots)
+  - [Node Parameter Snapshotting](#node-parameter-snapshotting)
+  - [Topic Recording Rules](#topic-recording-rules)
   - [Pattern Syntax](#pattern-syntax)
   - [QoS Settings](#qos-settings)
   - [Latched Topics](#latched-topics)
@@ -35,17 +41,23 @@ A ROS 2 **Humble** package that provides:
 | Feature | Details |
 |---------|---------|
 | **Glob topic matching** | Zenoh-style `*` (one segment) and `**` (any depth) |
+| **Type-based topic matching** | Match topics by message type (e.g. `sensor_msgs/msg/Image`) |
 | **Per-topic QoS** | `reliability`, `durability`, `depth` fully configurable |
-| **Latched topics** | Set `durability: transient_local` per topic |
+| **Latched topics** | Set `durability: transient_local` per topic; last value always included in snapshot |
 | **Max frequency** | Discard messages arriving faster than configured Hz |
-| **Fixed-duration buffer** | Always keeps the last *N* seconds (default 10 s) |
+| **Fixed-duration buffer** | Always keeps the last *N* seconds (default 60 s) |
 | **Memory efficiency** | Messages stored as raw serialised bytes; deque-based O(1) expiry |
 | **Dynamic type discovery** | Topic types resolved at runtime (like `ros2 bag record`) |
 | **On-demand snapshot** | `std_srvs/srv/Trigger` service writes the buffer to a rosbag |
-| **Configurable filename** | `{datetime}`, `{date}`, `{time}`, `{timestamp}` placeholders |
+| **Auto-snapshot** | Sliding-window automatic snapshots at a configurable interval |
+| **Configurable filename** | `{datetime}`, `{date}`, `{time}`, `{timestamp}`, `{trigger}` placeholders |
+| **Node parameter snapshotting** | Collects and stores active node parameters alongside each snapshot |
 | **REST API** | FastAPI server with OpenAPI docs at `/docs` |
-| **AI agent tools** | FastMCP server at `/mcp` (streamable-HTTP) |
-| **Image support** | `sensor_msgs/Image` and `CompressedImage` → PNG |
+| **AI agent tools** | FastMCP server at `/mcp` (streamable-HTTP); tools auto-derived from REST routes |
+| **Image support** | `sensor_msgs/Image` and `CompressedImage` → PNG (data-URL) |
+| **Sensor visualisation** | `sensor_msgs/LaserScan` → top-view PNG; `nav_msgs/Odometry` → trajectory PNG; `nav_msgs/OccupancyGrid` → map PNG |
+| **Topic statistics** | Per-topic message count, timestamps, and mean frequency |
+| **Log querying** | Filter `/rosout` entries by level, node name, or message text |
 | **Optional auth** | Bearer-token protection via `SNAPSHOT_SERVER_TOKEN` env var |
 
 ---
@@ -102,11 +114,15 @@ curl -H "Authorization: Bearer mysecrettoken" http://localhost:8001/snapshots
 
 # Get metadata of a specific snapshot
 curl -H "Authorization: Bearer mysecrettoken" \
-  http://localhost:8001/snapshots/snapshot_20240101_120000/metadata
+  http://localhost:8001/snapshots/snapshot_manual_20240101_120000/metadata
 
 # Get the 5 latest messages on a topic
 curl -H "Authorization: Bearer mysecrettoken" \
-  "http://localhost:8001/snapshots/snapshot_20240101_120000/messages?topic=/chatter&limit=5"
+  "http://localhost:8001/snapshots/snapshot_manual_20240101_120000/messages?topic=/chatter&limit=5"
+
+# Query WARN-and-above log entries from /rosout
+curl -H "Authorization: Bearer mysecrettoken" \
+  "http://localhost:8001/snapshots/snapshot_manual_20240101_120000/rosout?level=WARN"
 ```
 
 ### 4 – OpenAPI docs
@@ -117,19 +133,33 @@ Open `http://localhost:8001/docs` in your browser.
 
 ## Configuration Reference
 
-Copy `config/example_config.yaml` and adapt it:
+Copy `config/example_config.yaml` and adapt it.  A minimal example:
 
 ```yaml
 buffer:
-  duration_seconds: 10      # keep the last N seconds
+  duration_seconds: 60      # keep the last N seconds
 
 snapshot:
   directory: /tmp/rosbag_blackbox_snapshots
-  name_pattern: "snapshot_{datetime}"   # see placeholders below
+  name_pattern: "snapshot_{trigger}_{datetime}"   # see placeholders below
+
+auto_snapshot:
+  enabled: false
+  overlap_seconds: 10.0
+
+node_params:
+  patterns: []              # e.g. ["/**"] to capture every node's parameters
 
 topics:
+  - type: "sensor_msgs/msg/Image"
+    max_frequency: 1.0      # Hz; omit or 0 for unlimited
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 1
+
   - pattern: "/camera/**"
-    max_frequency: 10.0     # Hz; omit or 0 for unlimited
+    max_frequency: 10.0
     qos:
       reliability: best_effort
       durability: volatile
@@ -149,14 +179,104 @@ topics:
       depth: 10
 ```
 
+### Buffer Settings
+
+```yaml
+buffer:
+  duration_seconds: 60   # seconds of history to keep in memory (default: 60)
+```
+
+### Snapshot Output
+
+```yaml
+snapshot:
+  directory: /tmp/rosbag_blackbox_snapshots
+  name_pattern: "snapshot_{trigger}_{datetime}"
+```
+
 **Filename placeholders**
 
-| Placeholder | Example |
-|-------------|---------|
-| `{datetime}` | `20240101_120000` |
-| `{date}` | `20240101` |
-| `{time}` | `120000` |
-| `{timestamp}` | `1704110400` (Unix epoch) |
+| Placeholder | Example | Description |
+|-------------|---------|-------------|
+| `{datetime}` | `20240101_120000` | Date and time (`YYYYMMDD_HHMMSS`) |
+| `{date}` | `20240101` | Date only (`YYYYMMDD`) |
+| `{time}` | `120000` | Time only (`HHMMSS`) |
+| `{timestamp}` | `1704110400` | Unix epoch (integer seconds) |
+| `{trigger}` | `manual` or `auto` | How the snapshot was triggered |
+
+### Automatic Snapshots
+
+When enabled, the node writes a snapshot every
+`buffer_duration − overlap_seconds` seconds so that consecutive automatic
+snapshots share `overlap_seconds` of data:
+
+```yaml
+auto_snapshot:
+  enabled: true
+  overlap_seconds: 10.0   # seconds of shared data between consecutive snapshots
+```
+
+> **Note:** `overlap_seconds` must be less than `buffer.duration_seconds`;
+> otherwise auto-snapshot is disabled with a warning.
+
+### Node Parameter Snapshotting
+
+When `patterns` is non-empty, the node asynchronously collects the current
+ROS 2 parameters of every active node whose fully-qualified name matches at
+least one pattern and writes them to a `node_params.yaml` sidecar inside
+each snapshot directory:
+
+```yaml
+node_params:
+  patterns: ["/**"]               # zenoh-style glob patterns for node names
+  collection_interval_seconds: 30.0
+```
+
+The collected parameters can then be queried via the
+`/snapshots/{name}/node_params` REST endpoint.
+
+### Topic Recording Rules
+
+Topics can be matched by name **pattern**, message **type**, or both.
+Rules are evaluated **in order**; the first matching rule wins.
+If the `topics` section is empty (or the config file is omitted), all
+discovered topics are recorded with default QoS.
+
+```yaml
+topics:
+  # Match by message type (all topics publishing this type)
+  - type: "sensor_msgs/msg/Image"
+    max_frequency: 1.0
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 1
+
+  # Match by topic name pattern
+  - pattern: "/camera/**"
+    max_frequency: 10.0
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 1
+
+  # Match by both pattern AND type (both conditions must hold)
+  - pattern: "/robot/*"
+    type: "geometry_msgs/msg/Twist"
+    max_frequency: 10.0
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 5
+
+  # Catch-all fallback
+  - pattern: "/**"
+    max_frequency: 5.0
+    qos:
+      reliability: best_effort
+      durability: volatile
+      depth: 10
+```
 
 ### Pattern Syntax
 
@@ -170,7 +290,7 @@ The topic pattern follows **Zenoh selector** conventions:
 | `/*/scan` | `/lidar/scan`, `/front/scan`, … |
 | `/a/**/c` | `/a/c`, `/a/b/c`, `/a/b/d/c`, … |
 
-Rules are evaluated **in order**; the first matching pattern wins.
+The same syntax applies to `node_params.patterns` for node name matching.
 
 ### QoS Settings
 
@@ -192,6 +312,10 @@ qos:
   depth: 1
 ```
 
+The ring-buffer node keeps a separate cache for `transient_local` topics and
+always includes their most recent message in every snapshot, even after they
+have been evicted from the time-bounded ring buffer.
+
 > **Note:** If the publisher uses `volatile` durability and the subscriber
 > requests `transient_local`, ROS 2 will refuse the connection.  Always match
 > the publisher's durability setting.
@@ -222,6 +346,10 @@ rclpy.spin_until_future_complete(node, future)
 print(future.result().message)
 ```
 
+The `response.message` field contains the absolute path of the newly created
+rosbag directory.  Snapshots triggered this way use the `manual` value for
+the `{trigger}` filename placeholder.
+
 ---
 
 ## Snapshot Server
@@ -258,15 +386,24 @@ development but **strongly discouraged** in any network-accessible deployment.
 
 ### REST Endpoints
 
+All image/sensor render endpoints return a JSON object with a `data_url` key
+containing the image as a base64-encoded data URL
+(`data:image/png;base64,…`).
+
 | Method | Path | Description |
 |--------|------|-------------|
-| GET | `/health` | Liveness check |
-| GET | `/snapshots` | List all snapshots |
-| GET | `/snapshots/latest` | Path of the latest snapshot |
-| GET | `/snapshots/{name}/metadata` | Topics, counts, duration |
-| GET | `/snapshots/{name}/messages` | Extract messages (filterable) |
-| GET | `/snapshots/{name}/image/{topic}` | Latest image as PNG |
-| GET | `/snapshots/{name}/node_params` | List nodes with stored params |
+| GET | `/health` | Liveness check; returns server status and latest snapshot path |
+| GET | `/snapshots` | List all snapshot directory names |
+| GET | `/snapshots/latest` | Name of the most-recently modified snapshot |
+| GET | `/snapshots/{name}/metadata` | Topics, message counts, duration, time range |
+| GET | `/snapshots/{name}/messages` | Extract and deserialise messages (filterable) |
+| GET | `/snapshots/{name}/image/{topic}` | Nearest camera image as PNG data-URL |
+| GET | `/snapshots/{name}/laserscan/{topic}` | LaserScan rendered as top-view PNG data-URL |
+| GET | `/snapshots/{name}/trajectory/{topic}` | Odometry positions rendered as trajectory PNG data-URL |
+| GET | `/snapshots/{name}/map/{topic}` | OccupancyGrid rendered as map PNG data-URL |
+| GET | `/snapshots/{name}/topic_stats/{topic}` | Message count, timestamps, mean frequency |
+| GET | `/snapshots/{name}/rosout` | `/rosout` log entries (filterable by level, node, text) |
+| GET | `/snapshots/{name}/node_params` | List nodes with parameters stored in the snapshot |
 | GET | `/snapshots/{name}/node_params/{node}` | Parameters for a specific node |
 
 **Query parameters for `/messages`**
@@ -277,44 +414,73 @@ development but **strongly discouraged** in any network-accessible deployment.
 | `start` | float | Unix-time lower bound (seconds) |
 | `end` | float | Unix-time upper bound (seconds) |
 | `limit` | int | Max messages to return (default 100) |
-| `keys_only` | bool | Return only field names |
-| `truncate_arrays` | int | Truncate arrays to N elements |
+| `keys_only` | bool | Return only field names (no values) |
+| `truncate_arrays` | int | Truncate arrays to N elements (0 = no truncation) |
 
-### MCP Tools
+**Query parameters for `/rosout`**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `level` | string | Minimum level: `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL` (or numeric) |
+| `node` | string | Case-insensitive substring filter on the publishing node name |
+| `grep` | string | Case-insensitive substring filter on the log message text |
+| `start` | float | Unix-time lower bound (seconds) |
+| `end` | float | Unix-time upper bound (seconds) |
+| `limit` | int | Max log entries to return (default 200) |
+
+**Query parameters for `/image/{topic}`, `/laserscan/{topic}`, `/map/{topic}`**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `timestamp` | float | Unix-time in seconds; returns the nearest message (omit for first) |
+
+**Additional query parameters for `/trajectory/{topic}`**
 
-The FastMCP server at `/mcp` exposes the following tools to AI agents:
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `start` | float | Unix-time lower bound (seconds) |
+| `end` | float | Unix-time upper bound (seconds) |
+| `max_poses` | int | Maximum number of Odometry messages to include (default 2000) |
+| `img_size` | int | Edge length of the square output image in pixels (default 512) |
 
-| Tool | Description |
-|------|-------------|
-| `list_snapshots(directory)` | Discover snapshot files |
-| `select_snapshot(bag_path)` | Set the active snapshot |
-| `get_snapshot_metadata(bag_path)` | Topics, counts, duration |
-| `list_topics(bag_path)` | All topics with message counts |
-| `search_topics(pattern, bag_path)` | Find topics by substring |
-| `get_messages(topic, ...)` | Extract & deserialise messages |
-| `get_image(topic, ...)` | Return image as base64-PNG |
-| `list_param_nodes(bag_path)` | List nodes that have parameters in a snapshot |
-| `get_node_params(node_name, ...)` | Retrieve parameters for a specific node |
+### MCP Tools
 
-All tools default to the **latest** snapshot when `bag_path` is omitted.
+The FastMCP server at `/mcp` (streamable-HTTP) exposes **all FastAPI routes
+as MCP tools** via `FastMCP.from_fastapi`.  MCP tools are automatically
+generated from REST routes, eliminating the need for separate tool definitions.
+
+| Tool (derived from REST route) | Description |
+|-------------------------------|-------------|
+| `list_snapshots_rest` | List all snapshot directory names |
+| `get_latest_rest` | Name of the most-recently modified snapshot |
+| `get_metadata_rest` | Topics, counts, duration for a named snapshot |
+| `get_messages_rest` | Extract and deserialise messages (filterable) |
+| `get_image_rest` | Nearest camera image as base64 PNG data-URL |
+| `get_laserscan_rest` | LaserScan rendered as top-view PNG data-URL |
+| `get_trajectory_rest` | Odometry trajectory rendered as PNG data-URL |
+| `get_map_rest` | OccupancyGrid rendered as map PNG data-URL |
+| `get_topic_stats_rest` | Message count, timestamps, mean frequency for a topic |
+| `get_rosout_rest` | `/rosout` log entries filtered by level, node, or text |
+| `list_param_nodes_rest` | List nodes that have parameters in a snapshot |
+| `get_node_params_rest` | Parameters for a specific node in a snapshot |
 
 **Example agent interaction:**
 
 ```
-agent → list_snapshots()
-      ← [{"path": "/tmp/.../snapshot_20240101_120000", ...}]
+agent → list_snapshots_rest()
+      ← {"snapshots": ["snapshot_auto_20240101_115950", "snapshot_manual_20240101_120000"], "count": 2}
 
-agent → get_snapshot_metadata()
-      ← {"topics": [...], "total_messages": 1200, "duration_seconds": 10.0}
+agent → get_metadata_rest(bag_name="snapshot_manual_20240101_120000")
+      ← {"topics": [...], "total_messages": 1200, "duration_seconds": 60.0, ...}
 
-agent → list_topics()
-      ← [{"name": "/camera/image_raw", "type": "sensor_msgs/msg/Image", ...}]
+agent → get_messages_rest(bag_name="snapshot_manual_20240101_120000", topic="/tf", limit=3, truncate_arrays=5)
+      ← {"messages": [...], "count": 3}
 
-agent → get_image(topic="/camera/image_raw")
-      ← {"image_base64": "iVBORw0KGgoAAAANS...", "image_format": "PNG"}
+agent → get_image_rest(bag_name="snapshot_manual_20240101_120000", topic_path="camera/image_raw")
+      ← {"data_url": "data:image/png;base64,iVBORw0KGgoAAAANS..."}
 
-agent → get_messages(topic="/tf", max_messages=3, truncate_arrays=5)
-      ← {"messages": [...], "count": 3}
+agent → get_rosout_rest(bag_name="snapshot_manual_20240101_120000", level="WARN")
+      ← {"logs": [...], "count": 5}
 ```
 
 ---
@@ -325,7 +491,7 @@ agent → get_messages(topic="/tf", max_messages=3, truncate_arrays=5)
 ros2 launch rosbag_blackbox rosbag_blackbox.launch.py \
     config_file:=/path/to/config.yaml \
     snapshot_dir:=/data/snapshots \
-    buffer_duration:=30.0 \
+    buffer_duration:=60.0 \
     server_port:=8001
 ```
 
@@ -335,7 +501,7 @@ ros2 launch rosbag_blackbox rosbag_blackbox.launch.py \
 |----------|---------|-------------|
 | `config_file` | `share/…/example_config.yaml` | Path to YAML config |
 | `snapshot_dir` | `/tmp/rosbag_blackbox_snapshots` | Snapshot output dir |
-| `buffer_duration` | `10.0` | Ring buffer duration (seconds) |
+| `buffer_duration` | `60.0` | Ring buffer duration (seconds) |
 | `server_host` | `0.0.0.0` | Bind host for the API server |
 | `server_port` | `8001` | Port for the API server |
 
@@ -348,15 +514,18 @@ ros2 launch rosbag_blackbox rosbag_blackbox.launch.py \
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `config_file` | string | `''` | Path to YAML config file |
-| `buffer_duration` | double | `10.0` | Buffer duration in seconds |
-| `snapshot_dir` | string | `/tmp/…` | Snapshot output directory |
-| `snapshot_name_pattern` | string | `snapshot_{datetime}` | Filename pattern |
+| `buffer_duration` | double | `60.0` | Buffer duration in seconds |
+| `snapshot_dir` | string | `/tmp/rosbag_blackbox_snapshots` | Snapshot output directory |
+| `snapshot_name_pattern` | string | `snapshot_{trigger}_{datetime}` | Filename pattern |
+
+ROS parameters take precedence over the YAML config file when they differ
+from their default values.
 
 ### `snapshot_server`
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `snapshot_dir` | string | `/tmp/…` | Directory to search for bags |
+| `snapshot_dir` | string | `/tmp/rosbag_blackbox_snapshots` | Directory to search for bags |
 | `host` | string | `0.0.0.0` | Bind address |
 | `port` | int | `8001` | HTTP port |
 
@@ -381,19 +550,24 @@ ros2 launch rosbag_blackbox rosbag_blackbox.launch.py \
 │     │             │  ┌────────────▼──────────────┐    │      │
 │     │             │  │  save_snapshot service    │    │      │
 │     │             │  │  (std_srvs/Trigger)       │    │      │
+│     │             │  │  auto-snapshot timer      │    │      │
 │     │             │  └────────────┬──────────────┘    │      │
 │     │             └───────────────┼──────────────────┘      │
-│     │                             │ writes                   │
+│     │                             │ writes rosbag            │
+│     │                             │ + node_params.yaml       │
 │     │                             ▼                          │
 │     │                    ┌────────────────┐                  │
-│     │                    │  rosbag file   │                  │
-│     │                    │  (SQLite3)     │                  │
+│     │                    │  snapshot dir  │                  │
+│     │                    │  (rosbag +     │                  │
+│     │                    │  node_params)  │                  │
 │     │                    └────────┬───────┘                  │
 │                                  │ reads                     │
 │  ┌───────────────────────────────▼───────────────────────┐   │
 │  │  snapshot_server node                                  │   │
 │  │                                                        │   │
-│  │  FastAPI  ──→  REST endpoints (/snapshots, /image, …) │   │
+│  │  FastAPI  ──→  REST endpoints (/snapshots, /image,    │   │
+│  │                /laserscan, /trajectory, /map,          │   │
+│  │                /topic_stats, /rosout, /node_params, …) │   │
 │  │  FastMCP  ──→  /mcp  (streamable-HTTP)                │   │
 │  └────────────────────────────────────────────────────────┘   │
 └─────────────────────────────────────────────────────────────┘