sessions: add agent attempt comparisons - #335498
Draft
Megan Rogge (meganrogge) wants to merge 37 commits into
Draft
Megan Rogge (meganrogge) wants to merge 37 commits into
Megan Rogge (meganrogge) wants to merge 37 commits into
Conversation
Add a multi-harness comparison workflow with isolated child sessions, provider-owned shared-model resolution, visible judging, evidence review, synthesis, and explicit cleanup.\n\nFixes microsoft#335085\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Megan Rogge (meganrogge)
September 10, 2026 16:22
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Provenance restoration, file-overlap calculation, composer configuration forwarding, accessibility, layout, and test coverage need correction.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 7
New issues introduced by this change (8)
| Severity | Finding |
|---|---|
 |
src/vs/sessions/contrib/chat/browser/newChatInput.ts — Setting a visible label here does not resize the existing send control:… |
|
src/vs/sessions/contrib/chat/browser/newChatWidget.ts — The launch drops two values that this PR defines as shared: neither the composer's selected branch… |
|
src/vs/sessions/contrib/chat/browser/newChatWidget.ts — All coordinator/provider/worktree failures reach this catch, but they are only logged, so the user… |
|
src/vs/sessions/contrib/chat/browser/sessionTypePicker.ts — When a provider omits the optional pre-creation snapshot API, modelResolution is undefined and… |
|
src/vs/sessions/contrib/providers/copilotChatSessions/browser/copilotChatSessionsProvider.ts — This map only preserves provenance for the lifetime of the provider. After a window/provider reload… |
|
src/vs/sessions/contrib/sessionComparison/browser/sessionComparisonEditor.ts — This new focusable, information-rich editor has no editor-focused accessibility help provider,… |
|
src/vs/sessions/contrib/sessionComparison/browser/sessionComparisonEditor.ts — These keys are absolute paths inside each attempt's isolated worktree. The same repository-relative… |
 |
src/vs/sessions/services/sessions/browser/sessionComparisonService.ts — There are no focused tests for this new lifecycle service, despite neighboring Sessions browser… |
What changed in this PR
Adds provider-neutral multi-agent attempt comparisons to the Agents window.
Changes:
- Adds multi-harness comparison selection and shared-model validation.
- Orchestrates isolated attempts, judging, synthesis, persistence, and cleanup.
- Adds a comparison editor plus provider provenance/model-discovery support.
| File | Description |
|---|---|
src/vs/sessions/sessions.common.main.ts |
Loads comparison contributions and service. |
src/vs/sessions/SESSION_COMPARISONS.md |
Documents comparison architecture and lifecycle. |
src/vs/sessions/services/sessions/common/sessionsProvider.ts |
Extends provider creation and model-discovery contracts. |
src/vs/sessions/services/sessions/common/sessionsManagement.ts |
Adds parent-session creation provenance. |
src/vs/sessions/services/sessions/common/sessionComparison.ts |
Defines comparison models and service API. |
src/vs/sessions/services/sessions/browser/sessionsManagementService.ts |
Forwards session provenance to providers. |
src/vs/sessions/services/sessions/browser/sessionComparisonService.ts |
Implements orchestration, persistence, synthesis, and cleanup. |
src/vs/sessions/README.md |
Links the comparison specification. |
src/vs/sessions/contrib/sessionComparison/common/sessionComparison.ts |
Defines the comparison-opening command. |
src/vs/sessions/contrib/sessionComparison/browser/sessionComparisonTool.ts |
Adds the Judge verdict tool. |
src/vs/sessions/contrib/sessionComparison/browser/sessionComparisonEditorInput.ts |
Adds restorable comparison editor input. |
src/vs/sessions/contrib/sessionComparison/browser/sessionComparisonEditor.ts |
Renders evidence and comparison actions. |
src/vs/sessions/contrib/sessionComparison/browser/sessionComparison.contribution.ts |
Registers editor, tool, command, and setting. |
src/vs/sessions/contrib/sessionComparison/browser/media/sessionComparisonEditor.css |
Styles the comparison editor. |
src/vs/sessions/contrib/providers/copilotChatSessions/test/browser/copilotChatSessionsProvider.test.ts |
Tests Copilot provenance and model discovery. |
src/vs/sessions/contrib/providers/copilotChatSessions/browser/copilotChatSessionsProvider.ts |
Implements Copilot provenance and creation-time models. |
src/vs/sessions/contrib/providers/agentHost/test/browser/localAgentHostSessionsProvider.test.ts |
Tests Agent Host provenance and models. |
src/vs/sessions/contrib/providers/agentHost/browser/baseAgentHostSessionsProvider.ts |
Forwards provenance and exposes creation models. |
src/vs/sessions/contrib/chat/browser/sessionTypePicker.ts |
Adds comparison mode and multi-selection. |
src/vs/sessions/contrib/chat/browser/sessionsChatAccessibilityHelp.ts |
Documents comparison keyboard interaction. |
src/vs/sessions/contrib/chat/browser/newChatWidget.ts |
Launches comparisons from the composer. |
src/vs/sessions/contrib/chat/browser/newChatInput.ts |
Supports dynamic attempt-count send labels. |
build/lib/i18n.resources.json |
Registers comparison localization resources. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Preserve comparison configuration and provenance, make comparison evidence and controls accessible, normalize worktree changes, and add lifecycle regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Format comparison sources, preserve provider option shapes when provenance is absent, and compare persisted creation references through their public URI representation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Represent comparisons as uniquely identified attempts with provider-local model selection. Add a focused setup dialog, shared-context and usage disclosures, and keep synthesis explicit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Improve the parallel-attempt setup and comparison hierarchy, add structured Judge evidence and verdict tools, and record provider-neutral token usage with OTel correlation.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the Judge review every attempt and record validation provenance, then surface a recommendation-first result with explicit human review routing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Open live attempt sessions in an adaptive grid while work is active, then return to the standard comparison editor when all attempts are terminal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Load Judge instructions from a packaged prompt, open the comparison when judging completes, and present a winner-focused result with deterministic synthesis evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep comparison participants visually connected, preserve progress visibility, hide stale details on participant navigation, and place winner/synthesis actions at the end of completed results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Choose comparison grid columns from the number of active attempts so four sessions form a 2x2 grid, six form 3x2, and nine form 3x3. Keep phone layouts in a single column. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Replace the standalone comparison editor with a participant grid and a Judge-owned result card. Keep custom dialog controls in the keyboard focus loop so attempt agent and model selectors are operable without a mouse.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mplementation' into HEAD
Show Judge and synthesis before attempt sessions in both the list and tiled grid. Keep group connector decoration limited to implementation attempts.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Constrain the element measured by DomScrollableElement so additional attempt rows activate the custom scrollbar.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ensure setup groups, result actions, and compact comparison attempts expose names that match their visible presentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make attempt validation use authoritative worktrees, represent inapplicable validation explicitly, and allow rejected verdicts to be corrected. Repair the affected test and fixture infrastructure exposed by CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prevent verdict submission failures caused by models mistyping or combining participant and session UUIDs. Map manifest attempt numbers back to internal participant IDs when persisting the verdict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let the Judge identify semantic implementation decisions, allow users to choose an attempt per section, and pass the persisted plan to the isolated synthesis session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Show only the agent and model in synthesis approach selectors while keeping stable attempt numbers in the Judge contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Show comparison attempts by agent and model across session titles, pane headers, sidebar rows, and Judge results. Migrate untouched legacy generated titles while retaining stable attempt numbers in the Judge protocol. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Give parallel participant chats the full Sessions area by hiding the editor only after the comparison grid opens successfully. Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Keep the parallel-agent workflow discoverable when the selected workspace cannot create isolated worktrees, and rely on progress indicators instead of redundant working/completed labels. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve the latest synthesis and comparison-grid work while retaining persistent action discoverability and spinner-only attempt progress. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Format the comparison evidence mapping and make its worktree path assertion platform-native. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reject unusable comparison launches, preserve frozen permissions for Judge and synthesis sessions, restore telemetry coverage, and remove comparison APIs without production callers. Fix the comparison setup regression test so it exercises an eligible session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep concurrent attempt results widened to the persisted participant contract so launch-threshold checks typecheck. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ae987bcc-580a-4ea6-8789-828cf5d1d6fa
Fall back to the selected workspace repository branch when Agent Host creation config has not resolved yet, so Execute Parallel Agents opens its setup dialog instead of refocusing the workspace picker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…m:meganrogge/vscode into HEAD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sessions.chat.compareAgents.enabledexperimentUX
Comparison · Review readywhen the Judge submits a verdict.The prompt and attached context are frozen when the comparison launches. Judge recommendations remain advisory. Synthesis and worktree cleanup are always explicit, and no attempt is automatically applied or deleted.
Reviewer notes
Comparison records are provider-neutral and persisted in profile storage. Setup is represented as an array of uniquely identified attempts rather than a map keyed by harness, allowing different models from one harness or repeated runs of the same configuration.
Model identifiers remain provider-local and the UI displays provider-owned model labels. There is intentionally no equivalent-model-across-providers behavior.
Attempts, the Judge, and explicit synthesis are ordinary sibling sessions in one comparison group. The Judge is the automatic machine-review stage; Review Recommended Attempt is the explicit human verification stage. New verdicts distinguish attempt-reported validation, Judge-run validation, and unavailable evidence. Existing persisted verdicts remain compatible.
Validation
npm run transpile-clientnpm run typecheck-clientnpm run valid-layers-checkgit diff --checkFixes #335085
Architecture flow
flowchart LR Setup[Choose attempt agents/models<br/>and Judge] -->|Run in parallel| Attempts[Attempts<br/>isolated worktrees] Attempts -->|All terminal| Judge[Judge session<br/>review every diff] Judge -->|1. readAttemptComparison| Evidence[Bounded cross-attempt manifest] Evidence -.->|2. get_session_context<br/>for deeper transcript evidence| Judge Judge -.->|3. targeted tests/build/lint<br/>when evidence is missing| Validation[Validation evidence] Validation --> Judge Judge -->|4. completeAttemptComparison<br/>exactly once| Results[Recommendation<br/>provenance and conflicts] Results -->|Review Recommended Attempt| Review[Attempt session<br/>and Changes editor] Results -->|Optional| Synthesis[Synthesis session]Token usage stays out of Judge evidence; privacy-safe telemetry and OTel correlation are recorded separately.
Judge tools
readAttemptComparisonget_session_contextcompleteAttemptComparison