feat(metadata): draft/package lifecycle for publish-as-preview (overlay reads, teardown, discard, delete)

.changeset/ai-draft-aware-tools.md

@@ -0,0 +1,13 @@
+---
+'@objectstack/service-ai': patch
+---
+
+fix(ai): authoring tools can see their own drafts; blueprint surfaces the package to bind to
+
+Two gaps that broke the multi-step "build app → author a flow for it" path (found while verifying the new solution_design guardrail):
+
+1. **The agent couldn't discover its own draft objects.** `list_objects` / `list_metadata` read `getMetaItems` **active-only**, so a brand-new object the agent had just drafted (never published) was reported as "not found" when it then tried to author an approval flow against it. They now pass `previewDrafts: true`, overlaying pending drafts on the active list (older runtimes ignore the flag → stay active-only). `describe_metadata` was already draft-first.
+
+2. **The auto-authored flow had no package to bind to.** `apply_blueprint` already homes its artifacts in an app package, but its result only nested the id under `package`. It now also surfaces a top-level `packageId` and a `bindingHint` telling the agent to pass that `packageId` to `create_metadata` when it drafts follow-up automation (e.g. the approval flow) — so the flow lands in the app package instead of becoming an orphan draft.
+
+Together with the solution_design process guardrail, this makes the "model the data, then proactively draft the approval flow bound to the app" flow actually executable end-to-end.

.changeset/delete-drop-storage.md

@@ -0,0 +1,16 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/rest': patch
+---
+
+feat(metadata): optional storage teardown on delete so "publish to preview" leaves no orphan table
+
+Object storage was create-only: `publishMetaItem` creates a table (`ensureObjectStorage`) but nothing ever dropped one — `deleteMetaItem` only tombstones the metadata row, leaving the physical table behind. That made the pragmatic "publish an object just to preview it with real data, then discard if wrong" loop leave residue.
+
+Adds the inverse path, opt-in and guarded:
+
+- `engine.dropObjectSchema(name)` — inverse of `syncObjectSchema`; resolves the table name + driver and calls the driver's existing `dropTable` (DROP TABLE IF EXISTS / drop collection).
+- `deleteMetaItem({ …, dropStorage })` — when `true`, drops the object's physical table after the metadata is removed. **DESTRUCTIVE**, so it is gated: `object` type only (others have no table), `active` state only (drafts were never materialised), and never a `sys_`-prefixed platform table. Default `false` keeps delete non-destructive to data. Best-effort: a drop failure is logged, not thrown.
+- REST: `DELETE /meta/:type/:name?dropStorage=true` threads the flag.
+
+This makes "publish to preview → discard" cleanly reversible. Combined with the draft-overlay read mode, it backs the team's chosen approach: lean on publish (into a dev sandbox) for data-level confirmation rather than building a full draft-data preview, and make that publish safely undoable.

.changeset/draft-overlay-preview.md

@@ -0,0 +1,19 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/runtime': patch
+---
+
+feat(metadata): draft-overlay reads so an admin can render the console off pending drafts before publish
+
+ADR-0033's loop is `build (draft) → review → publish`, but "review" was only a JSON diff — the one thing that actually confirms an AI/hand-authored change (the rendered object page / kanban / form / nav) only existed *after* publish. That forces publishing unreviewed metadata just to look at it, defeating the draft gate.
+
+This adds a request-scoped **draft-overlay read mode** to the metadata resolution layer:
+
+- `getMetaItems({ …, previewDrafts })` — after the active overlay, overlays `state='draft'` rows on top (draft WINS on name collision; draft-only items surface too). Drafts are never hydrated into the process-wide SchemaRegistry.
+- `getMetaItem({ …, previewDrafts })` — non-strict: prefers a draft row if one exists, else falls back to the active value (unlike the strict `state:'draft'` mode, which 404s `no_draft`).
+- Every overlaid item is tagged `_draft: true` so the UI can badge it and show a "preview" banner.
+- The runtime HTTP dispatcher threads `?preview=draft` on `GET /metadata/:type` and `GET /metadata/:type/:name` into these reads.
+
+The same overlay also unblocks the AI authoring agent referencing its own just-drafted objects (a follow-up will point `list_metadata` at it). Admin gating of the `?preview=draft` flag is a deliberate follow-up step.
+
+Note: a brand-new draft object has no physical table until publish, so preview renders its *shape* (form/view/kanban/nav) but shows no data; field-additions to existing objects preview fully.

.changeset/package-discard-delete.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/runtime': patch
+---
+
+feat(packages): one-click discard-drafts and full delete for a package
+
+Two distinct package-level lifecycle operations, both built on the per-item delete primitive:
+
+- **`discardPackageDrafts(packageId)`** — drop every pending DRAFT bound to the package, reverting it to its last published baseline. NON-destructive: active/published metadata and physical tables are untouched. Use case: "I edited this app for a while and it turned out worse than before — abandon all my changes." Routes through the sys_metadata path (no metadata-service dependency, unlike the existing `POST /packages/:id/revert`, which 503s without a metadata service). REST: `POST /packages/:id/discard-drafts`.
+
+- **`deletePackage(packageId)`** — remove the ENTIRE package: every `sys_metadata` row (active + draft) and, by default, the physical table of each object it defined (DESTRUCTIVE). `keepData: true` removes metadata but preserves tables; the `sys_`-table guard still applies. Use case: "I don't want this package anymore." `DELETE /packages/:id` now performs this persisted removal in addition to the in-memory registry unregister it already did (previously it left AI/runtime packages' rows and tables behind); `?keepData=true` opts out of teardown.
+
+Drafts are deleted before active rows so each object's table is torn down exactly once. Per-item failures are collected without aborting the rest.

docs/notes/draft-overlay-preview-plan.md

@@ -0,0 +1,52 @@
+# Draft-overlay preview — implementation plan
+
+**Goal:** an admin flips a "preview drafts" switch and navigates the live console/app with
+every surface (object pages, list/kanban/form views, app nav, dashboards) rendered off
+**draft-overlaid** metadata — so AI/hand-authored changes can be *seen rendered* before publish,
+not just read as a JSON diff.
+
+**Unifying insight:** the same "draft-overlay read mode" serves BOTH
+- the human (preview mode renders drafts), and
+- the AI agent (`list_metadata` can see its own just-drafted objects — the Fix-2 gap that made
+  multi-step "build app → build flow" break because the flow step couldn't find draft objects).
+
+One foundation, two payoffs. Drafts become a first-class **renderable** state.
+
+## Hard limit (be honest)
+Brand-new draft objects have **no physical table until publish** (`ensureObjectStorage` runs on
+`publishMetaItem`). So preview of a *new* object shows **shape** (form layout, view columns, kanban
+groupBy, nav) but **no data / can't create records**. Field-additions to *existing* objects preview
+fully. Full data-preview for new objects needs draft-tables / a preview environment (`sys_metadata`
+already has `environment_id`) — deferred (ties to ADR-0027).
+
+## PRs
+
+### PR1 — backend draft-overlay foundation (framework)  ← THIS PR
+- `getMetaItems(request + previewDrafts?)`: after the active overlay, if `previewDrafts`, query
+  `state='draft'` rows (env-wide + org) and overlay them on top (draft WINS over active; draft-only
+  items appear). Mark each `_draft: true`. (protocol.ts ~1175-1234)
+- `getMetaItem(request + previewDrafts?)`: non-strict — draft if it exists, else fall back to active
+  (distinct from the existing strict `state:'draft'` which 404s). Mark `_draft: true`. (~1337)
+- Dispatcher `handleMetadata`: read `?preview=draft`, thread `previewDrafts` into both list + detail
+  reads. (http-dispatcher.ts ~947, ~1004)
+- Tests (protocol): draft overlays active; draft-only surfaces; `_draft` flag; getMetaItem fallback.
+- Changeset (objectql minor, runtime patch).
+- **No admin gate yet** — deferred to PR3 per product call (step 2).
+
+### PR1.5 — graceful no-table data path
+When `preview=draft` and a draft-only object has no table, the DATA query returns empty + a
+`draftNoTable` signal instead of "no such table". (data dispatcher)
+
+### PR2 — objectui frontend (sibling repo)
+- Admin-only "Preview drafts" toggle in the app shell; persistent "PREVIEW — drafts" banner.
+- Metadata client threads `?preview=draft` on all metadata reads when the toggle is on.
+- `_draft` badge on overlaid items; tolerate empty draft-object lists.
+
+### PR3 — admin gating (the user's explicit "step 2")
+Gate `preview=draft` to platform/org admins (reuse `isPlatformAdmin()`/`isActiveOrgAdmin()` from
+plugin-auth auth-manager.ts:961-1005). Non-admin flag → silently serve active (never leak drafts).
+
+### PR4 — AI discovery reuse (Fix 2)
+Point the AI `list_metadata`/`describe_metadata` tools at the same draft-overlay read so the agent
+can reference its own drafts; thread `packageId` into `create_metadata` so AI-authored flows bind to
+the app package (the orphan-flow bug found in verification).

packages/objectql/src/engine.ts

@@ -2361,6 +2361,23 @@ export class ObjectQL implements IDataEngine {
     await (driver as any).syncSchema(tableName, obj);
   }
 
+  /**
+   * Drop the physical storage (table/collection) backing an object — the
+   * inverse of {@link syncObjectSchema}. DESTRUCTIVE: deletes all rows in the
+   * table. Used by the protocol delete path when the caller explicitly opts
+   * into storage teardown (e.g. discarding an object that was published only
+   * to preview it). No-op when the object's driver does not expose `dropTable`.
+   * Resolves the physical table name from the registered definition, falling
+   * back to the bare name if the def was already removed.
+   */
+  async dropObjectSchema(objectName: string): Promise<void> {
+    const obj = this._registry.getObject(objectName) as any;
+    const driver = this.getDriverForObject(objectName);
+    if (!driver || typeof (driver as any).dropTable !== 'function') return;
+    const tableName = StorageNameMapping.resolveTableName(obj ?? ({ name: objectName } as any));
+    await (driver as any).dropTable(tableName);
+  }
+
   /**
    * Get a registered driver by datasource name.
    * Alias matching @objectql/core datasource() API.

packages/objectql/src/protocol-meta.test.ts

@@ -134,6 +134,77 @@ describe('ObjectStackProtocolImplementation - Metadata Persistence', () => {
         });
     });
 
+    describe('getMetaItems draft-overlay preview (ADR-0033)', () => {
+        const seedActiveAndDraft = () => mockEngine.find.mockImplementation((_t: string, opts: any) => {
+            const w = opts?.where ?? {};
+            if (w.type !== 'app') return Promise.resolve([]);
+            if (w.state === 'active') {
+                return Promise.resolve([
+                    { type: 'app', name: 'shared', state: 'active', metadata: JSON.stringify({ name: 'shared', label: 'Active' }) },
+                    { type: 'app', name: 'published_only', state: 'active', metadata: JSON.stringify({ name: 'published_only', label: 'Pub' }) },
+                ]);
+            }
+            if (w.state === 'draft') {
+                return Promise.resolve([
+                    { type: 'app', name: 'shared', state: 'draft', package_id: 'app.x', metadata: JSON.stringify({ name: 'shared', label: 'Draft' }) },
+                    { type: 'app', name: 'draft_only', state: 'draft', package_id: 'app.x', metadata: JSON.stringify({ name: 'draft_only', label: 'New' }) },
+                ]);
+            }
+            return Promise.resolve([]);
+        });
+
+        it('overlays drafts on active when previewDrafts is set (draft wins, draft-only surfaces, _draft tagged)', async () => {
+            seedActiveAndDraft();
+            const result = await protocol.getMetaItems({ type: 'app', previewDrafts: true });
+            const items = result.items as any[];
+            expect(items.map((i) => i.name).sort()).toEqual(['draft_only', 'published_only', 'shared']);
+            const shared = items.find((i) => i.name === 'shared');
+            expect(shared.label).toBe('Draft');     // draft wins over active
+            expect(shared._draft).toBe(true);
+            const draftOnly = items.find((i) => i.name === 'draft_only');
+            expect(draftOnly._draft).toBe(true);
+            expect(draftOnly._packageId).toBe('app.x');
+            expect(items.find((i) => i.name === 'published_only')._draft).toBeUndefined(); // active untouched
+        });
+
+        it('hides drafts by default (no previewDrafts)', async () => {
+            seedActiveAndDraft();
+            const result = await protocol.getMetaItems({ type: 'app' });
+            const items = result.items as any[];
+            expect(items.map((i) => i.name).sort()).toEqual(['published_only', 'shared']);
+            expect(items.find((i) => i.name === 'shared').label).toBe('Active');
+            expect(items.some((i) => i.name === 'draft_only')).toBe(false);
+        });
+    });
+
+    describe('getMetaItem draft-overlay preview (ADR-0033)', () => {
+        it('returns the draft when previewDrafts and a draft exists (_draft tagged, non-strict)', async () => {
+            mockEngine.findOne.mockImplementation((_t: string, opts: any) => {
+                const w = opts?.where ?? {};
+                if (w.state === 'draft' && w.name === 'lead') {
+                    return Promise.resolve({ type: 'object', name: 'lead', state: 'draft', package_id: 'app.x', metadata: JSON.stringify({ name: 'lead', label: 'Draft Lead' }) });
+                }
+                return Promise.resolve(null);
+            });
+            const res: any = await protocol.getMetaItem({ type: 'object', name: 'lead', previewDrafts: true });
+            expect(res.item.label).toBe('Draft Lead');
+            expect(res.item._draft).toBe(true);
+        });
+
+        it('falls back to active when previewDrafts but no draft exists (no no_draft 404)', async () => {
+            mockEngine.findOne.mockImplementation((_t: string, opts: any) => {
+                const w = opts?.where ?? {};
+                if (w.state === 'active' && w.name === 'lead') {
+                    return Promise.resolve({ type: 'object', name: 'lead', state: 'active', metadata: JSON.stringify({ name: 'lead', label: 'Active Lead' }) });
+                }
+                return Promise.resolve(null); // no draft row
+            });
+            const res: any = await protocol.getMetaItem({ type: 'object', name: 'lead', previewDrafts: true });
+            expect(res.item.label).toBe('Active Lead');
+            expect(res.item._draft).toBeUndefined();
+        });
+    });
+
     describe('saveMetaItem', () => {
         it('should throw when item data is missing', async () => {
             await expect(