Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 39 additions & 20 deletions docs/src/provider-platform-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -348,6 +348,25 @@ explicitly marks them safe for local-only use.

## Workstream A: Provider Platform Foundation

Track status: complete for [#47](https://github.com/AbdelStark/worldforge/issues/47).

Completion signals:

- Provider promotion rules cover `scaffold`, `experimental`, `beta`, and `stable` in the
[Provider Authoring Guide](./provider-authoring-guide.md), including profile metadata,
generated catalog updates, validation commands, and the current provider classification table.
- Runtime manifests are packaged under `src/worldforge/providers/runtime_manifests/`, validated by
tests, linked from provider docs, and used by health/configuration summaries without installing
optional runtimes.
- Capability conformance helpers exist for `predict`, `generate`, `transfer`, `reason`, `embed`,
`score`, and `policy`, plus provider-event redaction checks for safe fixture and injected-runtime
coverage.
- Optional live-smoke commands can write sanitized `run_manifest.json` evidence with command,
package, provider profile, capability, runtime manifest, input digest, event count, result digest,
and artifact-path summaries.
- Runtime pytest profiles keep default checkout validation deterministic while prepared hosts can
opt into `live`, `network`, `credentialed`, `gpu`, `robotics`, and provider-specific test runs.

### WF-PROV-001: Provider Promotion Gate

Type: provider platform
Expand All @@ -368,10 +387,10 @@ Scope:

Acceptance criteria:

- [ ] Promotion rules cover all current statuses: `scaffold`, `experimental`, `beta`, `stable`.
- [ ] Rules explain when to change provider profile metadata and generated catalog docs.
- [ ] Rules include exact local validation commands.
- [ ] Existing providers are classified against the new checklist without changing behavior.
- [x] Promotion rules cover all current statuses: `scaffold`, `experimental`, `beta`, `stable`.
- [x] Rules explain when to change provider profile metadata and generated catalog docs.
- [x] Rules include exact local validation commands.
- [x] Existing providers are classified against the new checklist without changing behavior.

Validation:

Expand Down Expand Up @@ -400,10 +419,10 @@ Scope:

Acceptance criteria:

- [ ] Manifest schema is documented and validated by tests.
- [ ] Manifests exist for `leworldmodel`, `lerobot`, `gr00t`, `cosmos`, and `runway`.
- [ ] Missing optional dependencies produce actionable health messages using manifest data.
- [ ] Docs link from provider pages to the relevant manifest.
- [x] Manifest schema is documented and validated by tests.
- [x] Manifests exist for `leworldmodel`, `lerobot`, `gr00t`, `cosmos`, and `runway`.
- [x] Missing optional dependencies produce actionable health messages using manifest data.
- [x] Docs link from provider pages to the relevant manifest.

Validation:

Expand Down Expand Up @@ -431,10 +450,10 @@ Scope:

Acceptance criteria:

- [ ] Each capability has a reusable conformance helper.
- [ ] Current provider tests call the helpers where applicable.
- [ ] The helpers can run against injected runtimes without credentials.
- [ ] The helpers do not use bare Python `assert` statements.
- [x] Each capability has a reusable conformance helper.
- [x] Current provider tests call the helpers where applicable.
- [x] The helpers can run against injected runtimes without credentials.
- [x] The helpers do not use bare Python `assert` statements.

Validation:

Expand Down Expand Up @@ -462,10 +481,10 @@ Scope:

Acceptance criteria:

- [ ] Live smoke commands can emit `run_manifest.json`.
- [ ] Manifest validation rejects secret-like metadata.
- [ ] Robotics showcase manifest links policy, score, replay, and report artifacts.
- [ ] Docs explain which artifacts are safe to attach to GitHub issues.
- [x] Live smoke commands can emit `run_manifest.json`.
- [x] Manifest validation rejects secret-like metadata.
- [x] Robotics showcase manifest links policy, score, replay, and report artifacts.
- [x] Docs explain which artifacts are safe to attach to GitHub issues.

Validation:

Expand Down Expand Up @@ -493,11 +512,11 @@ Scope:

Acceptance criteria:

- [ ] Live tests skip with clear reasons when runtime/env is missing.
- [ ] Prepared-host commands are documented for each real provider.
- [ ] Default CI remains deterministic and does not require credentials, network calls, GPUs, or
- [x] Live tests skip with clear reasons when runtime/env is missing.
- [x] Prepared-host commands are documented for each real provider.
- [x] Default CI remains deterministic and does not require credentials, network calls, GPUs, or
downloaded checkpoints.
- [ ] Live-smoke evidence can be attached to release notes or provider issues.
- [x] Live-smoke evidence can be attached to release notes or provider issues.

Validation:

Expand Down
54 changes: 54 additions & 0 deletions tests/test_docs_site.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,60 @@ def test_observability_roadmap_tracker_records_completion() -> None:
assert "request_id" in service_host


def test_provider_platform_foundation_roadmap_tracker_records_completion() -> None:
roadmap = (ROOT / "docs/src/provider-platform-roadmap.md").read_text(encoding="utf-8")
authoring = (ROOT / "docs/src/provider-authoring-guide.md").read_text(encoding="utf-8")
provider_index = (ROOT / "docs/src/providers/README.md").read_text(encoding="utf-8")
operations = (ROOT / "docs/src/operations.md").read_text(encoding="utf-8")
playbooks = (ROOT / "docs/src/playbooks.md").read_text(encoding="utf-8")

runtime_manifests = {
path.stem for path in (ROOT / "src/worldforge/providers/runtime_manifests").glob("*.json")
}

assert "Track status: complete for [#47]" in roadmap
for child in ("WF-PROV-001", "WF-PROV-002", "WF-PROV-003", "WF-PROV-004", "WF-PROV-005"):
assert child in roadmap

for completed_criterion in (
"Promotion rules cover all current statuses: `scaffold`, `experimental`, `beta`, `stable`.",
"Rules explain when to change provider profile metadata and generated catalog docs.",
"Manifest schema is documented and validated by tests.",
"Manifests exist for `leworldmodel`, `lerobot`, `gr00t`, `cosmos`, and `runway`.",
"Each capability has a reusable conformance helper.",
"The helpers do not use bare Python `assert` statements.",
"Live smoke commands can emit `run_manifest.json`.",
"Manifest validation rejects secret-like metadata.",
"Live tests skip with clear reasons when runtime/env is missing.",
"Prepared-host commands are documented for each real provider.",
"Live-smoke evidence can be attached to release notes or provider issues.",
):
assert f"- [x] {completed_criterion}" in roadmap

for status in ("`scaffold`", "`experimental`", "`beta`", "`stable`"):
assert status in authoring

for helper in (
"assert_predict_conformance",
"assert_generate_conformance",
"assert_transfer_conformance",
"assert_reason_conformance",
"assert_embed_conformance",
"assert_score_conformance",
"assert_policy_conformance",
"assert_provider_events_conform",
):
assert helper in authoring

assert {"leworldmodel", "lerobot", "gr00t", "cosmos", "runway"} <= runtime_manifests
assert "`src/worldforge/providers/runtime_manifests/`" in provider_index
assert "Optional live smoke entrypoints accept `--run-manifest <path>`" in provider_index

for marker in ("live", "network", "credentialed", "gpu", "robotics", "provider_profile"):
assert marker in operations
assert marker in playbooks


def test_reference_host_roadmap_tracker_records_completion() -> None:
roadmap = (ROOT / "docs/src/provider-platform-roadmap.md").read_text(encoding="utf-8")
examples = (ROOT / "docs/src/examples.md").read_text(encoding="utf-8")
Expand Down
Loading