Skip to content

AGENTS.md

Latest commit

 

History

History
352 lines (284 loc) · 13.5 KB

AGENTS.md

File metadata and controls

352 lines (284 loc) · 13.5 KB
Raw

AGENTS.md - AI Agent Guidance for VS Code AI Chat Analyzer

Overview

This file provides guidance and context to AI coding agents working on the vscode-ai-chat-analyzer extension. The extension analyzes AI chat responses and suggests updates to AGENTS.md files based on accuracy metrics.

Project Purpose

The VS Code AI Chat Analyzer is a Visual Studio Code extension designed to:

  • Analyze an AI chat for a single accuracy after completion
  • Parse and understand AGENTS.md files in workspace folders
  • Generate suggestions for improving AGENTS.md content based on the chat and the accuacy
  • Update AGENTS.md files based on analysis results (reinforce accurate behavior, mitigate inaccurate chats)
  • Help maintain high-quality AI agent documentation

Architecture

Current Flow (Interactive Chat Participant)

The extension provides an interactive chat participant for natural language feedback:

  1. User invokes @agentsfeedback with feedback text or slash command
  2. Chat participant extracts conversation history from previous turns
  3. References (#file, #selection) are resolved and included
  4. Command-specific prefixes are applied (for /learn, /stop, /remember)
  5. AI analyzer generates suggestions based on feedback and AGENTS.md context
  6. User selects which suggestions to apply via QuickPick UI
  7. Selected suggestions are appended to AGENTS.md

Alternative: Binary Feedback Flow

The extension also captures upvote/downvote feedback:

  1. User interacts with the chat participant and gives feedback (upvote/downvote)
  2. Feedback is converted to accuracy: Upvote → 0.9, Downvote → 0.3
  3. Single ChatResponse is created and passed through the analysis pipeline
  4. AI analyzer generates suggestions based on the single response and AGENTS.md context
  5. User selects which suggestions to apply via QuickPick UI
  6. Selected suggestions are appended to AGENTS.md

Core Components

1. Extension Entry Point (src/extension.ts)

Purpose: Registers VS Code commands, chat participant, and feedback handlers

Chat Participant: vscode-ai-chat-analyzer.feedback (invoked as @agentsfeedback)

  • Creates a chat participant for natural language feedback
  • Supports slash commands: /learn, /stop, /remember, /show
  • Reads conversation history from previous @agentsfeedback turns
  • Processes #file, #selection, and other VS Code references
  • Registers onDidReceiveFeedback handler for upvote/downvote events
  • Converts binary feedback to accuracy scores (Helpful = 0.9, Unhelpful = 0.3)
  • Creates a single ChatResponse from feedback metadata
  • Delegates to SuggestionProvider for analysis

Slash Commands:

  • /learn <instruction>: Reinforce positive patterns (prefixes with "POSITIVE PATTERN TO REINFORCE")
  • /stop <instruction>: Correct bad behaviors (prefixes with "NEGATIVE BEHAVIOR TO STOP")
  • /remember <instruction>: Add project-specific rules (prefixes with "PROJECT-SPECIFIC RULE TO REMEMBER")
  • /show: Display current AGENTS.md content without modification

Factory Functions:

  • createAnalyzer(): Returns ISuggestionAnalyzer based on aiChatAnalyzer.analyzer config
  • createSuggestionProvider(): Sets up provider with parser, updater, and analyzer

Manual Command: vscode-ai-chat-analyzer.analyzeChat

  • Alternative manual input method for testing
  • Prompts user for chat response text via input box
  • Allows selection of accuracy level
  • Useful for testing without actual chat interaction

2. Analyzers (src/analyzers/)

Purpose: Generate AGENTS.md suggestions using different LLM backends

Interface: ISuggestionAnalyzer

interface ISuggestionAnalyzer {
    analyzeSuggestions(
        analysisReport: { totalResponses: number; accurateResponses: number; accuracyRate: number; lowAccuracyResponses?: any[] },
        agentsData: any,
        chatResponses: ChatResponse[]
    ): Promise<Suggestion[]>;
}

Implementations:

  • CopilotSuggestionAnalyzer: Uses VS Code Language Model API with GitHub Copilot
  • LMStudioSuggestionAnalyzer: Uses local LM Studio server via HTTP
  • FakeSuggestionAnalyzer: Returns mock suggestions for testing

PromptBuilder (promptBuilder.ts):

  • Builds system and user prompts for LLM-based suggestion generation
  • Includes few-shot examples for better suggestion quality
  • Enforces token limits (~4096)

3. ChatAnalyzer (src/chatAnalyzer.ts)

Purpose: Calculates accuracy metrics from responses

Note: Currently processes single responses wrapped in an array. The aggregate metrics (totalResponses, accurateResponses, accuracyRate) have limited value with single responses but provide structure for potential future batch processing.

Usage (current single-response pattern):

const analyzer = new ChatAnalyzer();
const report = analyzer.analyzeResponses([
    { text: chatContent, accuracy: 0.9 }  // Single response
]);
// Returns: { totalResponses: 1, accurateResponses: 1, accuracyRate: 1, lowAccuracyResponses: [] }

4. SuggestionProvider (src/suggestionProvider.ts)

Purpose: Coordinates analysis and suggestion generation

Key Methods:

  • generateSuggestions(chatResponses): Runs ChatAnalyzer, parses AGENTS.md, calls analyzer
  • updateAgentsMd(suggestions): Applies selected suggestions via AgentsMdUpdater

5. SuggestionUI (src/suggestionUi.ts)

Purpose: User interface for selecting suggestions to apply

Key Functions:

  • showSuggestionSelection(suggestions): Multi-select QuickPick with Select All/Deselect All
  • showUpdateSuccess(count): Confirmation message after applying suggestions

6. AgentsMdParser (src/agentsMdParser.ts)

Purpose: Parses AGENTS.md files to extract agent information

Key Functionality:

  • Reads AGENTS.md file from filesystem
  • Parses markdown structure (headers as agent names)
  • Extracts agent responses/guidance from content
  • Provides structured access via getAgentsData()

7. AgentsMdUpdater (src/agentsMdUpdater.ts)

Purpose: Updates AGENTS.md files by appending new suggestions

Type Definitions

ChatResponse

interface ChatResponse {
    id: string;
    content: string;
    accuracy: number; // 0-1 scale (0.9 for upvote, 0.3 for downvote)
}

Suggestion

interface Suggestion {
    id: string;
    text: string;           // The actual suggestion content
    description?: string;   // Short description for UI
    reasoning?: string;     // Detailed reasoning shown in QuickPick detail
}

ISuggestionAnalyzer

interface ISuggestionAnalyzer {
    analyzeSuggestions(
        analysisReport: { totalResponses: number; accurateResponses: number; accuracyRate: number; lowAccuracyResponses?: any[] },
        agentsData: any,
        chatResponses: ChatResponse[]
    ): Promise<Suggestion[]>;
}

Legacy Types (still in codebase but underutilized)

interface AgentInfo {
    name: string;
    expectedResponses: string[];
    lastUpdated: Date;
}

interface AnalysisReport {
    agent: AgentInfo;
    responses: ChatResponse[];
    suggestions: string[];
}

Development Guidelines

When Working on This Extension

  1. Understanding Accuracy Thresholds:

    • High accuracy: ≥0.8
    • Medium accuracy: 0.5-0.8
    • Low accuracy: <0.5

  2. AGENTS.md File Structure:

    • Headers (#) denote agent names
    • Content under headers represents agent guidance
    • Parser extracts structure for programmatic access

  3. Error Handling:

    • Always check for workspace folder existence
    • Handle missing AGENTS.md gracefully
    • Provide clear error messages to users

  4. Testing:

    • Test suite located in test/suite/
    • Run tests with npm test
    • Coverage includes:
      • ChatAnalyzer accuracy calculations
      • Extension activation and registration
      • Chat participant registration
      • Slash command handling (/learn, /stop, /remember, /show)
      • Command prefix generation
      • Conversation history extraction
      • Reference processing
      • Feedback-to-accuracy conversion logic (upvote/downvote)
      • Command registration verification

Common Tasks

Adding New Analysis Metrics

  1. Update ChatAnalyzer.analyzeResponses() return type
  2. Add new calculation logic
  3. Update tests in test/suite/chatAnalyzer.test.ts
  4. Consider UI presentation in extension.ts

Modifying AGENTS.md Format

  1. Update AgentsMdParser.parseAgents() parsing logic
  2. Update Agent interface if structure changes
  3. Update AgentsMdUpdater to match new format
  4. Add tests for new format parsing

Adding New Commands

  1. Register in package.json under contributes.commands
  2. Add to activationEvents if needed
  3. Implement command handler in extension.ts
  4. Add corresponding tests

Dependencies

Runtime Dependencies

  • VS Code Extension API (^1.60.0)

Development Dependencies

  • TypeScript (^4.4.3)
  • Mocha (^11.7.5) - Test framework
  • Chai (^4.3.4) - Assertion library
  • @vscode/test-electron (^2.1.0) - VS Code testing utilities

Build & Test Commands

# Compile TypeScript
npm run compile

# Watch mode for development
npm run watch

# Run tests
npm test

# Package extension (requires @vscode/vsce)
vsce package

Known Limitations & Future Improvements

Current Limitations

  1. Feedback-based accuracy scoring (binary: upvote vs downvote)
  2. Basic AGENTS.md parsing (header-based only)
  3. Limited feedback data (only captures final vote, not intermediate states)
  4. No historical trend analysis across multiple sessions

Suggested Improvements

  1. Granular Feedback: ✅ IMPLEMENTED - Natural language feedback via @agentsfeedback chat participant
  2. Advanced Parsing: Support more complex AGENTS.md formats (lists, code blocks, etc.)
  3. Context Capture: ✅ IMPLEMENTED - Includes chat context (prompt, conversation history) in analysis
  4. Historical Tracking: Store analysis history and identify patterns over time
  5. Multi-file Support: Handle multiple AGENTS.md files in monorepos
  6. UI Enhancements: Add webview for rich analysis visualization and feedback history
  7. Smart Suggestions: Use AI to generate specific improvement recommendations
  8. A/B Testing: Compare different agent response strategies based on feedback

File Structure Quick Reference

vscode-ai-chat-analyzer/
├── src/
│   ├── extension.ts           # Main entry point, chat participant, commands
│   ├── chatAnalyzer.ts        # Response analysis and accuracy metrics
│   ├── agentsMdParser.ts      # AGENTS.md file parser
│   ├── agentsMdUpdater.ts     # AGENTS.md file updater
│   ├── suggestionProvider.ts  # Coordinates analysis pipeline
│   ├── suggestionUi.ts        # QuickPick UI for suggestion selection
│   ├── analyzers/
│   │   ├── copilotAnalyzer.ts       # GitHub Copilot LLM analyzer
│   │   ├── lmStudioAnalyzer.ts      # LM Studio local LLM analyzer
│   │   ├── FakeSuggestionAnalyzer.ts # Mock analyzer for testing
│   │   └── promptBuilder.ts         # Prompt construction for LLMs
│   └── types/
│       └── index.ts           # TypeScript type definitions
├── test/
│   ├── runTest.ts            # Test runner setup
│   ├── suite/
│   │   ├── index.ts          # Unit test suite configuration
│   │   ├── chatAnalyzer.test.ts    # ChatAnalyzer tests
│   │   ├── copilotAnalyzer.test.ts # Copilot analyzer tests
│   │   ├── lmStudioAnalyzer.test.ts # LM Studio analyzer tests
│   │   ├── promptBuilder.test.ts   # PromptBuilder tests
│   │   └── extension.test.ts       # Extension tests
│   └── integration/
│       ├── index.ts                    # Integration test configuration
│       ├── runIntegrationTest.ts       # Integration test runner
│       ├── analyzer.integration.test.ts # General analyzer tests
│       ├── copilot.integration.test.ts  # Copilot integration tests
│       ├── lmstudio.integration.test.ts # LM Studio integration tests
│       └── fake.integration.test.ts     # Fake analyzer integration tests
├── package.json              # Extension manifest
├── tsconfig.json            # TypeScript configuration
├── AGENTS.md                # This file - AI agent guidance
├── LOCAL_DEV.md             # Local development instructions
└── README.md                # User documentation

Questions to Consider

When working on this extension, consider these questions:

  1. Should aggregate metrics be removed from single-response flow?

    • Current: ChatAnalyzer calculates totalResponses, accuracyRate for arrays of 1
    • Consider simplifying for single-response focus

  2. How should different feedback types influence suggestions?

    • Upvote (0.9): Reinforce the behavior that worked
    • Downvote (0.3): Identify and correct problematic patterns

  3. Should the extension support batch analysis?

    • Historical accumulation of feedback over time
    • Periodic analysis of patterns across sessions

  4. What metadata should AGENTS.md track?

    • Timestamps, version history, accuracy trends?

  5. How to handle conflicts when updating AGENTS.md?

    • Manual review, auto-merge, or version control integration?

Contact & Contribution

  • Publisher: Keep-Social-Dev
  • Repository: vscode-ai-chat-analyzer
  • License: MIT

When contributing, ensure:

  • Tests pass (npm test)
  • Code compiles without errors (npm run compile)
  • Documentation is updated
  • AGENTS.md reflects any new patterns or components