Merge pull request #171 from AbdelStark/issue-141-adapter-author-workbench

Add adapter author workbench flow

Author: AbdelStark

CHANGELOG.md

@@ -9,6 +9,11 @@ releases may still include breaking changes when the public API needs to tighten
 
 ### Added
 
+- Added an adapter author workbench flow for provider promotion evidence. The non-Textual
+  workbench now handles catalog providers, scaffold providers, and the direct-construction
+  `jepa-wms` candidate; reports include runtime manifest status, fixture coverage, docs/catalog
+  drift, redaction checks, promotion gaps by target status, safe artifact references, and validation
+  commands, and `worldforge harness --flow workbench` exposes the same logic through TheWorldHarness.
 - Added cross-provider run comparisons for preserved eval and benchmark workspaces. `worldforge
   runs compare` now exports a shared JSON/Markdown/CSV model with provider rows, capability and
   operation context, fixture digest, suite version, budget status, event counts, missing evidence,

docs/src/provider-authoring-guide.md

@@ -48,14 +48,19 @@ Before opening a provider PR, run the non-TUI workbench from a clean checkout:
 ```bash
 uv run worldforge provider workbench mock
 uv run worldforge provider workbench <provider> --format json
+uv run worldforge provider workbench jepa-wms --format markdown
 uv run python scripts/generate_provider_docs.py --check
 ```
 
-The report lists the conformance helper required for every advertised capability, validates
-`tests/fixtures/providers/<provider>_*.json` playback files when they exist, links this guide, and
-prints a pasteable issue summary. It invokes only deterministic local providers by default. Use
-`--live` only on a prepared host when credentials, optional dependencies, injected runtimes, and
-runtime-owned artifacts are intentionally available.
+The report lists the conformance helper required for every advertised capability, planned
+capabilities for scaffold/candidate adapters, runtime manifest status, docs/catalog drift status,
+redaction checks, safe artifact references, and validation commands. It validates
+`tests/fixtures/providers/<provider>_*.json` playback files when they exist, including module-safe
+prefixes such as `jepa_wms_*.json` for direct-construction candidates. It also names missing
+promotion evidence by future status (`experimental`, `beta`, `stable`) so a scaffold gap is visible
+without turning into an accidental capability claim. It invokes only deterministic local providers
+by default. Use `--live` only on a prepared host when credentials, optional dependencies, injected
+runtimes, and runtime-owned artifacts are intentionally available.
 
 ## Adapter Decision Tree
 

docs/src/roadmap-continuation.md

@@ -693,10 +693,10 @@ Out of scope:
 
 Acceptance criteria:
 
-- [ ] Workbench can run against `mock` and at least one scaffold/candidate in a clean checkout.
-- [ ] Output names missing evidence by promotion status.
-- [ ] Markdown output includes validation commands and safe artifact references.
-- [ ] TUI and CLI workbench views use the same non-Textual flow logic.
+- [x] Workbench can run against `mock` and at least one scaffold/candidate in a clean checkout.
+- [x] Output names missing evidence by promotion status.
+- [x] Markdown output includes validation commands and safe artifact references.
+- [x] TUI and CLI workbench views use the same non-Textual flow logic.
 
 Validation:
 

docs/src/theworldharness.md

@@ -97,6 +97,7 @@ The diagnostics, eval, and benchmark screens map directly to non-TUI commands:
 uv run worldforge doctor --registered-only
 uv run worldforge provider list
 uv run worldforge provider workbench mock
+uv run worldforge harness --flow workbench
 uv run worldforge benchmark --provider mock --iterations 2 --format json
 uv run worldforge eval --suite planning --provider mock --format json
 ```
@@ -105,23 +106,32 @@ uv run worldforge eval --suite planning --provider mock --format json
 
 `worldforge provider workbench <provider>` is the checkout-safe adapter author loop behind the
 harness provider development workflow. It does not import Textual and does not make live provider
-calls unless `--live` is passed explicitly. The default report is designed to paste into GitHub
-issues: provider profile, required capability conformance helpers, fixture JSON status, docs/catalog
-drift hints, redaction-safe provider event status, and exact follow-up commands.
+calls unless `--live` is passed explicitly. The same non-Textual report model powers the
+`worldforge harness --flow workbench` TUI path. The default report is designed to paste into GitHub
+issues or PR descriptions: provider profile, target source, required capability conformance helpers,
+planned capability surface, runtime manifest status, fixture JSON status, docs/catalog drift hints,
+redaction-safe provider event status, promotion evidence grouped by future status, safe artifact
+references, and exact validation commands.
 
 ```bash
 uv run worldforge provider workbench mock
+uv run worldforge provider workbench jepa-wms --format markdown
+uv run worldforge provider workbench genie --format json
 uv run worldforge provider workbench runway --format json
 uv run worldforge provider workbench runway --live
+uv run worldforge harness --flow workbench
 ```
 
 For deterministic local providers such as `mock`, the workbench invokes the advertised capability
-helpers. For HTTP adapters it validates matching `tests/fixtures/providers/<provider>_*.json`
-playback files and lists the capability helpers that the provider test module must cover. For
-host-owned local runtimes such as LeRobot and LeWorldModel, the default path inspects profile,
-health, docs, and fixtures while leaving injected-runtime/live smoke execution to prepared hosts.
-Run `uv run python scripts/generate_provider_docs.py --check` before opening a provider PR so
-profile metadata and generated catalog tables stay in sync.
+helpers. For scaffold or direct-construction candidates such as `genie` and `jepa-wms`, it names
+missing evidence by promotion status instead of implying the provider is ready. For HTTP adapters it
+validates matching `tests/fixtures/providers/<provider>_*.json` or Python module-safe fixture
+prefixes such as `jepa_wms_*.json`, and lists the capability helpers that the provider test module
+must cover. For host-owned local runtimes such as LeRobot and LeWorldModel, the default path
+inspects profile, health, docs, runtime manifests, and fixtures while leaving injected-runtime/live
+smoke execution to prepared hosts. Run
+`uv run python scripts/generate_provider_docs.py --check` before opening a provider PR so profile
+metadata and generated catalog tables stay in sync.
 
 Completed checkout-safe flows also preserve a sanitized run workspace:
 

src/worldforge/harness/flows.py

@@ -67,6 +67,21 @@
             "mock provider benchmark matrix, and compare latency, throughput, and emitted events."
         ),
     ),
+    HarnessFlow(
+        id="workbench",
+        title="Adapter Author Workbench",
+        short_title="Workbench",
+        focus="adapter authoring evidence",
+        provider="Provider workbench",
+        capability="authoring",
+        command="uv run worldforge provider workbench mock",
+        accent="#f0a35e",
+        summary=(
+            "Run the checkout-safe provider workbench against the stable mock provider and the "
+            "direct-construction jepa-wms candidate, then collect promotion evidence, safe "
+            "artifact references, and validation commands."
+        ),
+    ),
 )
 
 
@@ -126,6 +141,50 @@ def _run_diagnostics_demo(*, state_dir: Path, emit: bool = False) -> JSONDict:
     return summary
 
 
+def _run_workbench_demo(*, state_dir: Path, emit: bool = False) -> JSONDict:
+    from worldforge.harness.workbench import (
+        provider_workbench_markdown,
+        provider_workbench_report,
+    )
+
+    reports = [
+        provider_workbench_report("mock", docs_root=Path.cwd()),
+        provider_workbench_report("jepa-wms", docs_root=Path.cwd()),
+    ]
+    providers = [str(report["provider"]) for report in reports]
+    passed = sum(1 for report in reports if report.get("status") == "passed")
+    safe_artifacts = [
+        artifact
+        for report in reports
+        for artifact in report.get("safe_artifacts", [])
+        if isinstance(artifact, dict)
+    ]
+    validation_commands = sorted(
+        {str(command) for report in reports for command in report.get("validation_commands", [])}
+    )
+    missing_by_provider = {
+        str(report["provider"]): report["promotion"]["missing_evidence_by_status"]
+        for report in reports
+    }
+    summary: JSONDict = {
+        "demo_kind": "provider_workbench",
+        "state_dir": str(state_dir),
+        "providers": providers,
+        "report_count": len(reports),
+        "passed_count": passed,
+        "failed_count": len(reports) - passed,
+        "reports": reports,
+        "safe_artifact_count": len(safe_artifacts),
+        "safe_artifacts": safe_artifacts,
+        "validation_commands": validation_commands,
+        "missing_evidence_by_provider": missing_by_provider,
+        "provider_events": [],
+    }
+    if emit:
+        print("\n\n".join(provider_workbench_markdown(report) for report in reports))
+    return summary
+
+
 # Demo modules import the optional-runtime provider classes at module scope, so
 # keep these imports lazy: loading the harness should not pull LeRobot/LeWorldModel
 # adapters into the base cold-start path.
@@ -145,6 +204,7 @@ def _run_lerobot_demo(**kwargs: object) -> JSONDict:
     "leworldmodel": _run_leworldmodel_demo,
     "lerobot": _run_lerobot_demo,
     "diagnostics": _run_diagnostics_demo,
+    "workbench": _run_workbench_demo,
 }
 
 
@@ -806,6 +866,43 @@ def _steps_for(flow_id: str, summary: JSONDict) -> tuple[HarnessStep, ...]:
                 "artifact=benchmark report json/markdown/csv",
             ),
         )
+    if flow_id == "workbench":
+        candidate_missing = summary["missing_evidence_by_provider"]["jepa-wms"]
+        return (
+            HarnessStep(
+                "Select authoring targets",
+                "Use one stable catalog provider and one direct-construction candidate.",
+                f"{', '.join(summary['providers'])} selected.",
+                "providers=mock,jepa-wms",
+            ),
+            HarnessStep(
+                "Run checkout-safe workbench",
+                "Invoke non-Textual provider_workbench_report without live provider calls.",
+                f"{summary['passed_count']}/{summary['report_count']} reports passed.",
+                "live=false",
+            ),
+            HarnessStep(
+                "Inspect promotion evidence",
+                "Group missing evidence by target promotion status.",
+                (
+                    "jepa-wms stable gaps: "
+                    f"{', '.join(candidate_missing.get('stable', [])) or 'none'}."
+                ),
+                "promotion=experimental,beta,stable",
+            ),
+            HarnessStep(
+                "Check runtime and fixtures",
+                "Read runtime manifest status and provider fixture coverage.",
+                f"{summary['safe_artifact_count']} safe artifact references collected.",
+                "fixtures=tests/fixtures/providers",
+            ),
+            HarnessStep(
+                "Render issue output",
+                "Preserve validation commands and safe artifact references for PRs or issues.",
+                f"{len(summary['validation_commands'])} validation commands listed.",
+                "format=markdown,json",
+            ),
+        )
     raise ValueError(f"unknown harness flow '{flow_id}'")
 
 
@@ -846,6 +943,36 @@ def _metrics_for(flow_id: str, summary: JSONDict) -> tuple[HarnessMetric, ...]:
                 "provider events captured during benchmark samples",
             ),
         )
+    if flow_id == "workbench":
+        candidate_missing = summary["missing_evidence_by_provider"]["jepa-wms"]
+        stable_missing = candidate_missing.get("stable", [])
+        return (
+            HarnessMetric(
+                "Targets",
+                str(summary["report_count"]),
+                ", ".join(summary["providers"]),
+            ),
+            HarnessMetric(
+                "Passed",
+                f"{summary['passed_count']}/{summary['report_count']}",
+                "checkout-safe reports",
+            ),
+            HarnessMetric(
+                "Candidate gaps",
+                str(len(stable_missing)),
+                ", ".join(stable_missing) or "none",
+            ),
+            HarnessMetric(
+                "Artifacts",
+                str(summary["safe_artifact_count"]),
+                "safe issue references",
+            ),
+            HarnessMetric(
+                "Commands",
+                str(len(summary["validation_commands"])),
+                "validation commands",
+            ),
+        )
 
     flow_label = "score" if flow_id == "leworldmodel" else "policy+score"
     return (
@@ -886,6 +1013,16 @@ def _transcript_for(flow_id: str, summary: JSONDict) -> tuple[str, ...]:
             f"benchmark_event_count: {summary['benchmark_event_count']}",
             f"commands: {' | '.join(summary['commands'])}",
         )
+    if flow_id == "workbench":
+        candidate_missing = summary["missing_evidence_by_provider"]["jepa-wms"]
+        return (
+            "flow: workbench",
+            f"providers: {', '.join(summary['providers'])}",
+            f"passed: {summary['passed_count']}/{summary['report_count']}",
+            f"jepa-wms_missing_stable: {', '.join(candidate_missing.get('stable', [])) or 'none'}",
+            f"safe_artifacts: {summary['safe_artifact_count']}",
+            f"validation_commands: {' | '.join(summary['validation_commands'])}",
+        )
 
     lines = [
         f"flow: {flow_id}",