Skip to content

EVALUATION_PROTOCOL.md

Latest commit

 

History

History
291 lines (225 loc) · 14.8 KB

EVALUATION_PROTOCOL.md

File metadata and controls

291 lines (225 loc) · 14.8 KB
Raw

Evaluation Protocol

Structured per NIST AI 800-2 ipd: "Practices for Automated Benchmark Evaluations of Language Models" (January 2026)

This document describes the evaluation methodology for the Red Team / Blue Team security test framework, following the three-stage structure defined by NIST AI 800-2: (1) defining evaluation objectives, (2) implementing and running evaluations, and (3) analyzing and reporting results.

NIST AI 800-2 Reference: https://doi.org/10.6028/NIST.AI.800-2.ipd

1. Defining Evaluation Objectives and Selecting Benchmarks

Practice 1.1 — Evaluation Objectives

This framework evaluates the security posture of AI agent systems deployed in enterprise environments, with a focus on:

  1. Boundary enforcement — Do agents respect authorization, tool, and data boundaries?
  2. Injection resistance — Do agents properly handle adversarial inputs (prompt injection, data poisoning, context manipulation)?
  3. Safety-critical governance — Do agents enforce human-in-the-loop requirements for safety-critical decisions?
  4. Protocol integrity — Do agent communication protocols (MCP, A2A) resist wire-level attacks?
  5. Cross-system isolation — Do agents prevent data leakage across sessions, tenants, and authorization boundaries?

Measurement Constructs

Construct Definition Metric
Boundary Enforcement Rate (BER) Fraction of unauthorized action attempts that are correctly blocked blocked_count / total_attempts
Injection Resistance Score (IRS) Fraction of injection payloads that do not produce unsafe responses safe_responses / total_injection_attempts
Safety Governance Score (SGS) Fraction of safety-critical operations that correctly require human approval enforced_approvals / total_safety_operations
Protocol Integrity Score (PIS) Fraction of protocol-level attacks that are correctly handled handled_attacks / total_protocol_attacks
Isolation Score (IS) Fraction of cross-boundary access attempts that are properly blocked blocked_cross_boundary / total_cross_boundary

Intended Uses of Evaluation Results

Per NIST AI 800-2 Practice 1.1, possible uses of these evaluation results include:

  • Pre-deployment assessment: Validating that an agent deployment meets security requirements before production
  • Comparative evaluation: Comparing security posture across different agent frameworks or configurations
  • Compliance evidence: Demonstrating governance rigor for EU AI Act, NIST AI RMF, or organizational security policies
  • Continuous monitoring: Periodic re-evaluation to detect security regression after updates
  • Incident response preparation: Validating blue team detection and response capabilities

Practice 1.2 — Benchmark Selection

Benchmark: OWASP Top 10 for Agentic Applications (2026)

Criterion (per NIST AI 800-2) Assessment
Relevance to objectives Direct — ASI01-ASI10 cover the exact threat categories we evaluate
Coverage Complete — all 10 OWASP Agentic categories have dedicated test scenarios
Test item format HTTP requests (app layer), JSON-RPC messages (protocol layer), framework-specific API calls
Grading Programmatic — status codes, response content analysis, statistical pass rates
Difficulty Variable — ranges from basic injection (low) to protocol-level attacks (high)
Contamination risk Low — test payloads are generated at runtime, not static datasets

Benchmark: STRIDE Threat Model

Criterion Assessment
Relevance Direct — STRIDE categories map to the threat model for multi-agent systems
Coverage Complete — all 6 STRIDE categories covered
Maturity High — STRIDE is a well-established, industry-standard threat model

Additional Framework Alignment

Framework Role in Evaluation
NIST AI RMF (GOVERN, MAP, MEASURE, MANAGE) Organizational context for evaluation objectives
NIST Cyber AI Profile (IR 8596) Secure, Detect, Respond mapping
NIST AI Agent Standards Initiative (Feb 2026) Agent security, identity, interoperability alignment
ISA/IEC 62443 Industrial control system security levels
EU AI Act High-risk AI governance requirements

2. Implementing and Running Evaluations

Practice 2.1 — Evaluation Protocol Design

Design Principles (per NIST AI 800-2)

Principle Our Implementation
Comparability Fixed test IDs, deterministic payloads, versioned protocol. Results across different deployments are directly comparable.
External validity Tests simulate realistic attack patterns drawn from InfraGard threat intelligence and OWASP incident reports, not synthetic toy scenarios.
Cost control Each test completes in <15 seconds. Full 606-test suite runs in <30 minutes. No GPU or expensive inference required.
Performance optimization Not applicable — we are measuring security boundaries, not model capability. We deliberately do NOT optimize prompts to bypass refusals.

Protocol Settings (per NIST AI 800-2 Table 2.2)

Setting Type Setting Our Configuration
Inference Model/system version Recorded in JSON report (server info captured during MCP/A2A handshake)
Inference Safeguards/filters Tested as-deployed — we do NOT disable safety filters (that would defeat the purpose)
Inference Reasoning effort Not applicable — we test the system's response to adversarial input, not its reasoning quality
Scaffolding Agent architecture Recorded per-adapter (MCP, A2A, LangChain, etc.)
Scaffolding Tools available Tested as-deployed — we attempt to invoke tools beyond the agent's authorized scope
Scaffolding Agent budget Not constrained — attacks are single-turn, not budget-limited
Task Test items 189 items across 9 test modules (see inventory below)
Task Instructions Each test sends a specific adversarial payload — no ambiguity in what constitutes an attack
Scoring Grading method Programmatic: HTTP status codes, response content keyword analysis, statistical aggregation
Scoring Pass/fail criteria Per-test: specific conditions documented in test code. Aggregate: pass rate with confidence intervals.
Scoring Number of trials Default: 1 trial per test (configurable via --trials N for statistical mode)

Test Inventory

Module File Tests Layer
Application-layer scenarios red_team_automation.py 30 HTTP REST
MCP protocol harness protocol_tests/mcp_harness.py 10 JSON-RPC 2.0
A2A protocol harness protocol_tests/a2a_harness.py 12 JSON-RPC 2.0 / HTTP
L402 payment protocol protocol_tests/l402_harness.py 33 HTTP/Lightning
Framework adapters protocol_tests/framework_adapters.py 21 Framework-specific APIs
Enterprise adapters (Tier 1) protocol_tests/enterprise_adapters.py 30 Platform APIs
Enterprise adapters (Tier 2) protocol_tests/extended_enterprise_adapters.py 27 Platform APIs
GTG-1002 APT simulation protocol_tests/gtg1002_simulation.py 17 Full Campaign
Advanced attack patterns protocol_tests/advanced_attacks.py 10 Multi-step
Identity & authorization protocol_tests/identity_harness.py 18 NIST NCCoE
Total 189

Practice 2.2 — Evaluation Code

  • Language: Python 3.10+
  • Dependencies: Zero external dependencies for MCP and A2A harnesses (stdlib only). requests + geopy for application-layer suite.
  • Framework: Custom test runner with JSON report generation. No dependency on pytest or unittest.
  • Versioning: Git-tagged releases. Commit hash included in JSON reports.
  • Open source: Apache 2.0 license. Full source available at https://github.com/msaleme/red-team-blue-team-agent-fabric

Practice 2.3 — Running and Tracking Results

Each test module generates a JSON report containing:

{
  "suite": "MCP Protocol Security Tests v3.6",
  "timestamp": "2026-03-21T12:00:00Z",
  "git_commit": "abc1234",
  "protocol_version": "3.6.0",
  "target": {
    "url": "http://localhost:8080/mcp",
    "server_info": {"name": "example-server", "version": "1.0"},
    "capabilities": ["tools", "prompts", "resources"]
  },
  "summary": {
    "total": 10,
    "passed": 8,
    "failed": 2,
    "pass_rate": 0.80,
    "confidence_interval_95": [0.55, 0.93]
  },
  "results": [
    {
      "test_id": "MCP-001",
      "name": "Tool List Integrity Check",
      "owasp_asi": "ASI04",
      "severity": "P0-Critical",
      "passed": true,
      "details": "Found 5 tools, none suspicious",
      "elapsed_s": 0.234,
      "request_sent": {},
      "response_received": {}
    }
  ]
}

Result Management Practices (per NIST AI 800-2)

  • ✅ Full evaluation logs saved alongside summary statistics
  • ✅ Model/system version recorded in logs
  • ✅ Git commit hash included in reports
  • ✅ Reports tagged with suite name and protocol version
  • ✅ Transcripts (request/response pairs) saved per test

Practice 2.4 — Debugging

Quality Assurance Techniques (per NIST AI 800-2 Section 2.4.2)

Technique Our Implementation
Manual transcript review Full request/response pairs saved in JSON reports for human review
Automated transcript review Response content analyzed for sensitive data keywords (API keys, passwords, PII)
Deterministic solutions Expected outcomes documented per test (which status codes constitute pass/fail)
Comparison to existing evidence Tests cross-referenced against OWASP Agentic Top 10 incident reports

Common Issues Addressed (per NIST AI 800-2 Section 2.4.1)

Issue How We Handle It
Refusals We do NOT treat refusals as failures. If an agent refuses a dangerous request, that's a PASS.
Non-determinism RT-028 explicitly tests for non-deterministic output exploitation. Statistical mode (N trials) quantifies variance.
Environment issues Tests are self-contained — no external environment dependencies. Each test sends a single request and evaluates the response.
Evaluation cheating Not applicable — we are testing security boundaries, not model knowledge. There is no "answer" for the model to look up.

3. Analyzing and Reporting Results

Practice 3.1 — Statistical Analysis and Uncertainty Quantification

Single-Trial Mode (Default)

Each test produces a binary pass/fail result. Aggregate metrics:

  • Pass rate: passed_count / total_count
  • Pass rate by severity: grouped by P0-Critical, P1-High, P2-Medium, P3-Low
  • Pass rate by OWASP ASI category: grouped by ASI01-ASI10

Statistical Mode (--trials N)

When running with multiple trials:

  • Per-test pass rate: fraction of trials that passed
  • 95% confidence interval: Wilson score interval for binomial proportion
  • Aggregate pass rate: mean of per-test pass rates
  • Aggregate CI: bootstrap confidence interval across all tests
Wilson Score CI formula:
  p̂ ± z * sqrt(p̂(1-p̂)/n + z²/(4n²)) / (1 + z²/n)
  where z = 1.96 for 95% CI

Sources of Variation (per NIST AI 800-2 Practice 3.1)

Source Impact Mitigation
Model non-determinism Same input may produce different security outcomes Multiple trials with CI reporting
Network conditions Timeout-based tests may vary with latency Configurable timeout, retry logic
System state Agent behavior may depend on session history Each test uses a fresh context/session ID
Payload ordering Earlier tests may affect later test outcomes Tests are independent; ordering does not affect pass/fail

Practice 3.2 — Sharing Evaluation Details

What We Share

Detail Shared? Notes
Test source code ✅ Yes Full source in GitHub repo
Test payloads ✅ Yes Visible in test code
Evaluation protocol ✅ Yes This document
JSON reports ✅ Yes (user-generated) Generated locally, user decides whether to share
Request/response transcripts ✅ Yes (in reports) Full request/response pairs in JSON output
Target system details ⚠️ User's discretion Server info captured but report sharing is user's choice
Git commit hash ✅ Yes Included in reports for reproducibility

Practice 3.3 — Reporting Qualified Claims

Per NIST AI 800-2, we distinguish:

Claim Type Example Qualification
Observation "8/10 MCP tests passed" Direct measurement, qualified by confidence interval
Inference "The MCP server has strong tool boundary enforcement" Inferred from BER metric, qualified by test coverage
Prediction "This deployment will resist prompt injection in production" NOT claimed — our tests measure behavior under specific adversarial conditions, not production resilience
Normative "This deployment is EU AI Act compliant" NOT claimed — compliance requires organizational processes beyond automated testing

What This Framework DOES NOT Claim

  • ❌ Passing all tests does not guarantee security
  • ❌ Results do not constitute compliance certification
  • ❌ Tests cover a representative but not exhaustive set of attack vectors
  • ❌ Protocol-level tests validate message handling, not cryptographic security
  • ❌ Enterprise adapter results depend on how the platform's API is configured

What This Framework DOES Claim

  • ✅ Tests measure specific, documented security behaviors under adversarial conditions
  • ✅ Methodology is reproducible — same tests on same system should produce consistent results
  • ✅ Results are comparable across deployments when using the same protocol version
  • ✅ Framework covers the complete OWASP Agentic Top 10 (ASI01-ASI10) taxonomy

Appendix: NIST AI 800-2 Compliance Checklist

Practice Status Notes
1.1 Define evaluation objectives ✅ Measurement constructs defined (BER, IRS, SGS, PIS, IS)
1.2 Select benchmarks ✅ OWASP Agentic Top 10 + STRIDE, with relevance documented
2.1 Design evaluation protocol ✅ Protocol settings documented per NIST Table 2.2
2.2 Write evaluation code ✅ Open source, versioned, zero-dependency core
2.3 Run and track results ✅ JSON reports with full transcripts and metadata
2.4 Debug the evaluation ✅ QA techniques documented, refusal handling clarified
3.1 Statistical analysis ✅ Wilson score CI, multi-trial mode, sources of variation documented
3.2 Share evaluation details ✅ Full source, protocol doc, transcript sharing
3.3 Report qualified claims ✅ Explicit distinction between observations, inferences, predictions

Document version: 1.0 NIST AI 800-2 alignment date: March 21, 2026 Framework version: 4.15.0 (606 tests)