Skip to content

feat: agent-driven uploads (images, traces, files)#30

Merged
benvinegar merged 4 commits into
mainfrom
claude/determined-gates-rf9s72
Jun 16, 2026
Merged

feat: agent-driven uploads (images, traces, files)#30
benvinegar merged 4 commits into
mainfrom
claude/determined-gates-rf9s72

Conversation

@benvinegar

Copy link
Copy Markdown
Member

Agents can now push binary and structured artifacts — images, screenshots, agent traces, arbitrary files — over all three tiers (CLI, MCP, raw HTTP), and the viewer surfaces them natively.

Summary

This implements a complete asset upload system with two new part kinds (image and trace) that agents can reference by ID. Assets are stored separately from surfaces to avoid bloating the 2 MB surface limit with binary data, and include reference-aware LRU eviction to stay within a per-board budget.

Key changes

  • New Asset entity in server/types.ts: stores uploaded blobs with metadata (kind, contentType, byteLength, filename). Scoped to sessions for cascade deletion.
  • Two new part kinds:
    • ImagePart: references an uploaded image by assetId, rendered natively by the viewer as a trusted <img> (no iframe).
    • TracePart: renders a step timeline beside the surface. Steps can travel inline (small, structured) or live in an uploaded JSON/JSONL file offered for download.
  • HTTP API (POST /api/assets, GET /a/:id): accepts raw bytes with metadata in query params, or base64-JSON bodies. Serves assets with security-conscious headers (raster images inline, SVG/JSON/text as attachments to prevent script execution).
  • Storage:
    • SqlStore: new assets table with BLOB data column (33% smaller than base64, no decode-on-serve).
    • JsonFileStore: new assets[] entry; bytes persist/load as base64 (JSON limitation), live as Uint8Array in memory.
  • Reference-aware LRU eviction: when a new upload would exceed MAX_BOARD_ASSET_BYTES (~2 GB), putAsset evicts oldest-lastAccessedAt-first, but partitioned so unreferenced assets go first. Assets still referenced by live surface parts are only evicted as a last resort. lastAccessedAt is bumped on every serve, protecting raw-URL embeds that referencedAssetIds() can't see.
  • CLI commands: sideshow upload, sideshow image, sideshow trace to push assets and publish them as surfaces. sideshow publish --image to add an image part to an html surface.
  • MCP support: upload_asset tool in the MCP schema; coerceParts validates image and trace parts from tool args.
  • Viewer components: ImagePart.tsx and TracePart.tsx render assets natively. Missing/evicted assets show a clear placeholder rather than broken images.
  • Security: CSP updated to allow origin in img-src and media-src so uploaded assets at <origin>/a/:id embed by URL. Sandboxed iframes still can't access them directly (opaque origin), but <img> and fetch work.

Implementation details

  • Uint8Array is the standard-lib type for bytes across the Store interface and in memory; base64 is only an edge encoding (HTTP request bodies, MCP args, JsonFileStore on-disk JSON).
  • partsByteLength() extended to count image/trace part refs (small); asset bytes are bounded separately by the board budget.
  • firstHtml/stripParts remain correct — image/trace parts are structured data, passed through untouched like diff.
  • Both stores pass the extended test/storeContract.ts suite, which now includes asset tests.
  • E2E tests cover image embedding, trace rendering (inline and uploaded), and asset eviction scenarios.

https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY

claude added 4 commits June 16, 2026 17:03
Explore supporting agent-pushed images, traces, and files across the
CLI/MCP/HTTP tiers: assets as a first-class entity stored apart from
surfaces, new image/trace part kinds rendered natively by the viewer,
upload/serve endpoints, the CSP change needed to embed served assets by
URL, storage/migration for both stores, and tests/risks.

https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
Settle the four open questions: blob bytes in DO SQLite as BLOB (no R2,
Uint8Array through the Store, base64 only at edges); 5 MB per-asset cap +
~2 GB board budget with reference-aware LRU eviction and an "evicted"
viewer placeholder so it never silently breaks a card; session-scoped
asset lifetime; and a content-type serving policy (raster inline, all
else attachment + nosniff, html/unknown coerced to octet-stream).

https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
Agents can push images, traces, and files across all three tiers and
surface them in the viewer.

- Assets are a first-class entity stored apart from surfaces: bytes ride
  as Uint8Array through the Store, BLOB in SqlStore, base64 only at the
  HTTP/MCP edges and JsonFileStore's on-disk JSON. Per-asset cap 5 MB;
  board budget enforced by reference-aware LRU eviction (selectEvictions)
  that evicts unreferenced assets before live ones and keeps served assets
  warm. Session-scoped lifetime, cascading on session delete. Both stores
  pass an extended store contract; the node:sqlite shim now binds blobs.
- New part kinds: `image` (assetId, rendered natively) and `trace` (inline
  step timeline and/or an uploaded JSON/JSONL file with a download link),
  rendered by new ImagePart/TracePart viewer components with an "evicted"
  placeholder for missing assets.
- Endpoints: POST /api/assets (raw bytes or base64-JSON envelope, detected
  by reading the body once) and GET /a/:id, which serves raster images
  inline and everything else as attachment + nosniff (octet-stream for
  html/unknown) so an upload can't execute as a same-origin document.
- Tiers: upload_asset MCP tool (stdio + HTTP), image/trace in the parts
  schema, and CLI `upload` / `image` / `trace` / `publish --image`.
- The surface CSP now allows the server's own origin so uploaded assets
  embed by URL (<img src="/a/:id">) on http localhost and deployments.
- Docs: design doc marked implemented; design guide, AGENTS map, and
  changelog updated.

https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
- test/assets.test.ts: runnable unit tests for the reference-aware LRU
  eviction (selectEvictions: under-budget no-op, oldest-first, unreferenced
  before referenced, referenced only as last resort), collectAssetIds,
  partsByteLength on image/trace, and coerceParts for image/trace parts.
- e2e/uploads.spec.ts: image part renders an <img> that actually loads;
  trace timeline expands its detail; an uploaded trace file shows a
  download link and renders its steps; an image embeds by URL inside an
  html part under the widened CSP; a missing asset shows the placeholder.
- e2e/fixtures.ts: upload / publishParts helpers + a tiny PNG fixture.

https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
@benvinegar benvinegar merged commit 8720a57 into main Jun 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants