|
| 1 | +# e2e smoke test and Go-vs-Python parity checklist |
| 2 | + |
| 3 | +## 1. Purpose |
| 4 | + |
| 5 | +This directory is the evidence gate for cutting over from the Python `cde-plugin/bin/runcode` to the Go `runcode` binary. It contains two things: |
| 6 | + |
| 7 | +- An opt-in live smoke test (`smoke_test.go`, build-tagged `e2e`) that drives a real workspace through a full lifecycle and asserts key behaviors. |
| 8 | +- This parity checklist (sections 5-7) that tells the operator how to verify the Go binary matches the Python binary's observable contract before flipping production traffic. |
| 9 | + |
| 10 | +The cutover itself is out of scope here. Pass this checklist first. |
| 11 | + |
| 12 | +## 2. WARNING |
| 13 | + |
| 14 | +**Running the smoke test creates and then deletes a real, billed workspace.** |
| 15 | + |
| 16 | +- With `RUNCODE_API_BASE` unset the test runs against **production**. Point it at a non-prod backend by setting `RUNCODE_API_BASE`. |
| 17 | +- The test sets `RUNCODE_CACHE_HOME` to an isolated temp dir for the duration of the run. It will NOT touch or corrupt your real `~/.cache/runcode` session or stored token. |
| 18 | +- The delete step runs via `t.Cleanup`, so the workspace is removed even if an intermediate step fails. A very early panic (e.g., build failure) could leave it orphaned; check the dashboard if the run crashes hard. |
| 19 | + |
| 20 | +## 3. How to run |
| 21 | + |
| 22 | +``` |
| 23 | +RUNCODE_TOKEN=<real-token> go test -tags e2e ./e2e/ -v |
| 24 | +``` |
| 25 | + |
| 26 | +Run from the `runcode-cli/` directory. |
| 27 | + |
| 28 | +**Required:** `RUNCODE_TOKEN` must be set to a valid API token. Without it the test skips immediately, which is why the default `go test ./...` never runs it. |
| 29 | + |
| 30 | +**Optional:** `RUNCODE_API_BASE` overrides the API endpoint (default: prod). Set it to target a staging or local backend. |
| 31 | + |
| 32 | +The `forward` probe sub-step needs `python3` on the workspace. If it is absent the sub-step is soft-skipped (logged, not failed). Everything else still runs. |
| 33 | + |
| 34 | +## 4. What the smoke test covers |
| 35 | + |
| 36 | +Steps run in order; delete is guaranteed via `t.Cleanup`: |
| 37 | + |
| 38 | +1. **create** - `create --size tiny --json` - waits for SSH-ready, parses `workspace_id` + asserts `created=true` and `attached=true`. |
| 39 | +2. **exec** - `exec -- echo <marker>` - asserts the marker string appears in stdout. |
| 40 | +3. **write + get round-trip** - writes a binary-safe file (includes non-ASCII byte) to the remote workdir, reads it back with `get --out`, asserts byte-for-byte equality. |
| 41 | +4. **forward + probe** - starts `python3 -m http.server` on the workspace at port 8099, opens a local forward, HTTP-probes it with retries; cancels the forward on exit (soft-skipped if python3 absent). |
| 42 | +5. **delete** (cleanup) - `delete <id> --yes --json` - asserts `deleted=true`. |
| 43 | + |
| 44 | +## 5. Parity checklist |
| 45 | + |
| 46 | +**Method:** run the Go binary and the Python `cde-plugin/bin/runcode` with identical arguments. Where the command supports `--json`, add it to both invocations. After each run, diff three things: |
| 47 | + |
| 48 | +- **error `code` strings** (the `"code"` field in JSON error output) |
| 49 | +- **process exit codes** |
| 50 | +- **the set of JSON keys** in the top-level response object |
| 51 | + |
| 52 | +All three must match on every command except the intentional differences listed in section 6. After the final fix wave, the error `code` strings and exit codes match Python everywhere (soft errors exit 1, operational errors exit 2), and `list --json` matches Python's bare-array-minus-`provider` shape. |
| 53 | + |
| 54 | +### Table A - offline / error-path parity (no workspace needed; run these first) |
| 55 | + |
| 56 | +| # | Command | Go invocation | Python invocation | Diff (codes/exit/keys) | Result | |
| 57 | +|---|---------|---------------|-------------------|------------------------|--------| |
| 58 | +| 1 | `--version` | `runcode --version` | `cde-plugin/bin/runcode --version` | Version STRING differs by build; assert both print a version line and exit 0 | | |
| 59 | +| 2 | `login` / no-token shape | `runcode logout && runcode list --json` (no token present) | same | both should produce `no_token` error code, same exit | | |
| 60 | +| 3 | `list --json` | `runcode list --json` | `cde-plugin/bin/runcode list --json` | bare array; per-element keys match (Go lists all by default — see §6.5; Python may show fewer rows but the same shape) | | |
| 61 | +| 4 | `status --json` | `runcode status --json` | `cde-plugin/bin/runcode status --json` | keys/codes/exit | | |
| 62 | +| 5 | `current --json` (no attachment) | `runcode current --json` | `cde-plugin/bin/runcode current --json` | both report "none"/not_found shape | | |
| 63 | +| 6 | `exec` with no attached workspace | `runcode exec -- echo hi` | `cde-plugin/bin/runcode exec -- echo hi` | both `not_found`, exit non-zero | | |
| 64 | +| 7 | `delete` without `--yes` | `runcode delete somews` | `cde-plugin/bin/runcode delete somews` | both `confirm_required`, exit non-zero, ZERO destructive API calls | | |
| 65 | +| 8 | `get` with no attachment | `runcode get /nope` | `cde-plugin/bin/runcode get /nope` | both `not_found` | | |
| 66 | +| 9 | `port-forward` bad port | `runcode port-forward notaport` | `cde-plugin/bin/runcode forward notaport` | both `bad_request` (Go `forward` alias also works) | | |
| 67 | +| 10 | `doctor --json` | `runcode doctor --json` | `cde-plugin/bin/runcode doctor --json` | **intentional difference** - see section 6; do NOT count as failure | | |
| 68 | +| 11 | `clean` | `runcode clean` | `cde-plugin/bin/runcode clean` | both preserve the token; compare behavior | | |
| 69 | + |
| 70 | +### Table B - live-lifecycle parity (needs a workspace; same args, `--json`) |
| 71 | + |
| 72 | +Run both binaries against the same workspace (or sequential workspaces of the same size). Diff codes/exit/keys for each. |
| 73 | + |
| 74 | +| # | Command | Go invocation | Python invocation | Diff (codes/exit/keys) | Result | |
| 75 | +|---|---------|---------------|-------------------|------------------------|--------| |
| 76 | +| 1 | `create` | `runcode create --size tiny --json` | `cde-plugin/bin/runcode create --size tiny --json` | keys/codes/exit | | |
| 77 | +| 2 | `connect` | `runcode connect <id> --json` | `cde-plugin/bin/runcode connect <id> --json` | keys/codes/exit | | |
| 78 | +| 3 | `exec` | `runcode exec -- echo hi` | `cde-plugin/bin/runcode exec -- echo hi` | keys/codes/exit | | |
| 79 | +| 4 | `ssh` (was `run`) | `runcode ssh <id> -- ls` | `cde-plugin/bin/runcode run <id> -- ls` | keys/codes/exit (Go `run` alias also works) | | |
| 80 | +| 5 | `context` | `runcode context --json` | `cde-plugin/bin/runcode context --json` | keys/codes/exit | | |
| 81 | +| 6 | `write` | `runcode write test.txt --file /tmp/x` | `cde-plugin/bin/runcode write test.txt --file /tmp/x` | keys/codes/exit | | |
| 82 | +| 7 | `put` | `runcode put /tmp/x test.txt` | `cde-plugin/bin/runcode put /tmp/x test.txt` | keys/codes/exit | | |
| 83 | +| 8 | `get` | `runcode get test.txt --out /tmp/y` | `cde-plugin/bin/runcode get test.txt --out /tmp/y` | keys/codes/exit | | |
| 84 | +| 9 | `port-forward` (was `forward`) | `runcode port-forward 8080 --local 9090` | `cde-plugin/bin/runcode forward 8080 --local 9090` | keys/codes/exit | | |
| 85 | +| 10 | `port-forward --cancel` | `runcode port-forward 8080 --local 9090 --cancel` | `cde-plugin/bin/runcode forward 8080 --local 9090 --cancel` | keys/codes/exit | | |
| 86 | +| 11 | `disconnect` | `runcode disconnect --json` | `cde-plugin/bin/runcode disconnect --json` | keys/codes/exit | | |
| 87 | +| 12 | `stop` | `runcode stop <id> --json` | `cde-plugin/bin/runcode stop <id> --json` | keys/codes/exit | | |
| 88 | +| 13 | `delete --yes` | `runcode delete <id> --yes --json` | `cde-plugin/bin/runcode delete <id> --yes --json` | keys/codes/exit | | |
| 89 | + |
| 90 | +## 6. Intentional differences |
| 91 | + |
| 92 | +These are the owner-approved, deliberate deviations from the Python `cde-plugin/bin/runcode`. They are NOT regressions; every other observable behavior must match. The error `code` strings and process exit codes match Python on every command (soft/user errors exit 1, operational errors exit 2). |
| 93 | + |
| 94 | +1. **`doctor --json` check-name set.** The Go `doctor` checks `token`, `api-base`, `cache`, `platform`, `status-line`, `auth`; Python checks for a Python interpreter, `ssh`, and `ssh-keygen` on `PATH`. The Go CLI speaks SSH natively (`golang.org/x/crypto/ssh`) and never shells out, so those binaries are irrelevant; the `cache`-writability probe replaces them. The `checks[].name` set therefore differs by design. |
| 95 | +2. **Human / progress text goes to stderr** (Python prints it to stdout). `--json` output is on stdout in both, so machine consumers see identical streams — parity is preserved for the path that matters. |
| 96 | +3. **`status` / `current` / `disconnect` accept `--json`** (Python's argparse for those rejects the flag). A superset, so any Python invocation still works. |
| 97 | +4. **`exec` with no attached workspace returns error code `not_found`** (Python returns a generic `error`). More specific, exit code unchanged. |
| 98 | +5. **`list` lists ALL workspaces by default and accepts `--all` as a no-op alias** (Python defaults to SSH-connectable-only and uses `--all` to widen). Scripts passing `--all` keep working; `list --json` otherwise emits the same bare array (each element minus the internal `provider` field, plus a computed `connectable`). |
| 99 | +6. **Loopback HTTP allowance is narrower:** only `127.0.0.1` / `::1` / `localhost` are accepted over plain http, vs Python's whole `127.0.0.0/8`. Stricter, never looser. |
| 100 | +7. **(Known limitation, not yet fixed)** `context` / `write` / `put` / `get` do not enforce a per-call timeout (Python caps reads at 30s and writes at 120s). An unreachable box is still bounded by the 20s SSH handshake, so calls cannot hang indefinitely. |
| 101 | +8. **Command surface aligned to Coder/Gitpod conventions** (the Go CLI is the *replacement* for the Python `cde-plugin`, so it is free to improve once cut over). All Python names still work as hidden aliases, so existing scripts and muscle memory are unaffected: |
| 102 | + - **`ssh <ws> [-- cmd]`** is the human verb (bare → interactive shell, `-- cmd` → one-shot). It absorbs Python's **`run`** and **`shell`**, both kept as hidden aliases. The agent-facing **`exec`** (targets the *attached* workspace, no positional) is unchanged. |
| 103 | + - **`port-forward <port>`** renames Python's **`forward`** (kept as a hidden alias). `port-forward` is unambiguous and matches `coder port-forward`. |
| 104 | + - **`start <ws>`** (NEW, no Python equivalent) powers a stopped box on and waits for `running` *without* attaching — the inverse of `stop`; `--no-wait` returns immediately. (`connect --start` remains the power-on-AND-attach combo.) |
| 105 | + - **`open <ws>`** (NEW, no Python equivalent) opens the workspace browser IDE, or prints the URL with `--print` (and auto-falls-back to printing on a headless box). It reads the cached session's `web_url` when one is fresh, else mints to learn the current URL. URL is validated to absolute http/https before being handed to the OS opener (security: never a `file://`/option-like string to `xdg-open`/`open`/`rundll32`, always via argv, never a shell). |
| 106 | + |
| 107 | +## 7. Results log |
| 108 | + |
| 109 | +Fill this in after the operator run: |
| 110 | + |
| 111 | +``` |
| 112 | +Date: ____________________ |
| 113 | +Operator: ____________________ |
| 114 | +Backend: prod | ____________________ (RUNCODE_API_BASE) |
| 115 | +Workspace id: ____________________ |
| 116 | +Region/size: ____________________ |
| 117 | +Go binary ver: ____________________ |
| 118 | +
|
| 119 | +Smoke lifecycle: |
| 120 | + create ........ PASS / FAIL notes: |
| 121 | + exec .......... PASS / FAIL notes: |
| 122 | + write+get ..... PASS / FAIL notes: |
| 123 | + forward+probe . PASS / FAIL / SKIPPED(python3 absent) notes: |
| 124 | + delete ........ PASS / FAIL notes: |
| 125 | +
|
| 126 | +Parity (Go vs Python --json): codes / exit / keys |
| 127 | + Table A offline: PASS / FAIL notes: |
| 128 | + Table B live: PASS / FAIL notes: |
| 129 | +
|
| 130 | +Overall: PASS / FAIL |
| 131 | +``` |
0 commit comments