feat: agent-driven uploads (images, traces, files)#30
Merged
Conversation
Explore supporting agent-pushed images, traces, and files across the CLI/MCP/HTTP tiers: assets as a first-class entity stored apart from surfaces, new image/trace part kinds rendered natively by the viewer, upload/serve endpoints, the CSP change needed to embed served assets by URL, storage/migration for both stores, and tests/risks. https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
Settle the four open questions: blob bytes in DO SQLite as BLOB (no R2, Uint8Array through the Store, base64 only at edges); 5 MB per-asset cap + ~2 GB board budget with reference-aware LRU eviction and an "evicted" viewer placeholder so it never silently breaks a card; session-scoped asset lifetime; and a content-type serving policy (raster inline, all else attachment + nosniff, html/unknown coerced to octet-stream). https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
Agents can push images, traces, and files across all three tiers and surface them in the viewer. - Assets are a first-class entity stored apart from surfaces: bytes ride as Uint8Array through the Store, BLOB in SqlStore, base64 only at the HTTP/MCP edges and JsonFileStore's on-disk JSON. Per-asset cap 5 MB; board budget enforced by reference-aware LRU eviction (selectEvictions) that evicts unreferenced assets before live ones and keeps served assets warm. Session-scoped lifetime, cascading on session delete. Both stores pass an extended store contract; the node:sqlite shim now binds blobs. - New part kinds: `image` (assetId, rendered natively) and `trace` (inline step timeline and/or an uploaded JSON/JSONL file with a download link), rendered by new ImagePart/TracePart viewer components with an "evicted" placeholder for missing assets. - Endpoints: POST /api/assets (raw bytes or base64-JSON envelope, detected by reading the body once) and GET /a/:id, which serves raster images inline and everything else as attachment + nosniff (octet-stream for html/unknown) so an upload can't execute as a same-origin document. - Tiers: upload_asset MCP tool (stdio + HTTP), image/trace in the parts schema, and CLI `upload` / `image` / `trace` / `publish --image`. - The surface CSP now allows the server's own origin so uploaded assets embed by URL (<img src="/a/:id">) on http localhost and deployments. - Docs: design doc marked implemented; design guide, AGENTS map, and changelog updated. https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
- test/assets.test.ts: runnable unit tests for the reference-aware LRU eviction (selectEvictions: under-budget no-op, oldest-first, unreferenced before referenced, referenced only as last resort), collectAssetIds, partsByteLength on image/trace, and coerceParts for image/trace parts. - e2e/uploads.spec.ts: image part renders an <img> that actually loads; trace timeline expands its detail; an uploaded trace file shows a download link and renders its steps; an image embeds by URL inside an html part under the widened CSP; a missing asset shows the placeholder. - e2e/fixtures.ts: upload / publishParts helpers + a tiny PNG fixture. https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents can now push binary and structured artifacts — images, screenshots, agent traces, arbitrary files — over all three tiers (CLI, MCP, raw HTTP), and the viewer surfaces them natively.
Summary
This implements a complete asset upload system with two new part kinds (
imageandtrace) that agents can reference by ID. Assets are stored separately from surfaces to avoid bloating the 2 MB surface limit with binary data, and include reference-aware LRU eviction to stay within a per-board budget.Key changes
Assetentity inserver/types.ts: stores uploaded blobs with metadata (kind, contentType, byteLength, filename). Scoped to sessions for cascade deletion.ImagePart: references an uploaded image byassetId, rendered natively by the viewer as a trusted<img>(no iframe).TracePart: renders a step timeline beside the surface. Steps can travel inline (small, structured) or live in an uploaded JSON/JSONL file offered for download.POST /api/assets,GET /a/:id): accepts raw bytes with metadata in query params, or base64-JSON bodies. Serves assets with security-conscious headers (raster images inline, SVG/JSON/text as attachments to prevent script execution).SqlStore: newassetstable withBLOB datacolumn (33% smaller than base64, no decode-on-serve).JsonFileStore: newassets[]entry; bytes persist/load as base64 (JSON limitation), live asUint8Arrayin memory.MAX_BOARD_ASSET_BYTES(~2 GB),putAssetevicts oldest-lastAccessedAt-first, but partitioned so unreferenced assets go first. Assets still referenced by live surface parts are only evicted as a last resort.lastAccessedAtis bumped on every serve, protecting raw-URL embeds thatreferencedAssetIds()can't see.sideshow upload,sideshow image,sideshow traceto push assets and publish them as surfaces.sideshow publish --imageto add an image part to an html surface.upload_assettool in the MCP schema;coercePartsvalidatesimageandtraceparts from tool args.ImagePart.tsxandTracePart.tsxrender assets natively. Missing/evicted assets show a clear placeholder rather than broken images.origininimg-srcandmedia-srcso uploaded assets at<origin>/a/:idembed by URL. Sandboxed iframes still can't access them directly (opaque origin), but<img>and fetch work.Implementation details
Uint8Arrayis the standard-lib type for bytes across theStoreinterface and in memory; base64 is only an edge encoding (HTTP request bodies, MCP args,JsonFileStoreon-disk JSON).partsByteLength()extended to countimage/tracepart refs (small); asset bytes are bounded separately by the board budget.firstHtml/stripPartsremain correct — image/trace parts are structured data, passed through untouched likediff.test/storeContract.tssuite, which now includes asset tests.https://claude.ai/code/session_01XWa7rTcBdXdEHg3Px9rXGY