[Spec drafted by @brandonjackson 30 March 2026]

Problem Statement

Sandboxes do not properly support Collections. When running a Workflow in a Sandbox, users can only access production collections data.

Current Behavior

When a sandbox is provisioned from a parent project, collections are not created or linked in the sandbox. Workflows that reference collections by name (e.g., collections.get('my-collection')) will fail with "not found" errors in the sandbox because no collection with that name exists there.

Additionally, collection names are currently globally unique across the entire Lightning instance (enforced by unique_index(:collections, [:name])), which means a sandbox cannot even create a collection with the same name as one in its parent project.

Impact

Services teams are unwilling to use sandboxes on projects that use collections. This is a blocker for sandboxes adoption and must be resolved before public release.

Requirements

Auto-create empty collections in sandbox on provision

• When a sandbox is forked, create empty Collection records matching each parent collection's name
• This ensures job code referencing collections.get('my-collection') resolves correctly in the sandbox
• Collections should be empty (no items copied) — sandbox is for testing, not data duplication

Scope collection name uniqueness to project (not global)

• Enable both the main branch and multiple sandboxes to all have collections with the same name in the same project (rather than enforcing global uniqueness)

Sandbox merge merges configuration not data

• When merging a sandbox back into its parent, collection data is never merged / never overwrites parent data
• New collections created in sandbox are created in parent sandbox as empty collections
• Existing collections deleted in sandbox are deleted in parent project

Implementation Details

• Update collection uniqueness constraints so that the name is only unique within a project.
• Presumably this means adding a project key to the database and migrate all collections to include a matching key
• Accept a version flag on the collections API. Probably via header.
• Update the v1 version to search for a collection by name, and return 409 if more than one match is found
• Add a new v2 collections API which uses the path collections/:project-id/:name
• Pass the owning project id to the options of the Run sent to the worker (as options.project_id)
• On sandbox fork, initialise new empty collections and add to the new project
• On sandbox merge, add/remove collections to the target project (but do not sync data)

Out of Scope

There are several ideas which we could/should do later which are out of scope:

• Support sharing collections across environments, eg namespace syntax like main::my-collection to explicitly reference a collection from a different environment
• Collection data seeding, Option to copy a subset of collection items when provisioning a sandbox (for test data)

Archive: Original Ticket

Reminder to ensure that collections will work in sandboxes

I haven't thought about this enough: but we need to ensure that collections are namespaced and isolated for each sandbox.

So in job code I do: collections.get('my-collection'), but the main project and staging sandbox access different variations of that.

So collection names are not unique within a workspace.

When looking up a credential by name we probably need to know what sandbox you're in, so that we can find correct collection to use