Skip to content

Latest commit

 

History

History
179 lines (138 loc) · 8.49 KB

File metadata and controls

179 lines (138 loc) · 8.49 KB

The Decision Governance Checklist

A standalone, citable checklist for evaluating whether an autonomous AI agent's governance layer can actually hold under pressure from the agent it governs. First published as Section 7 of Decision Governance Benchmark: Executable Behavioral Tests for Autonomous AI Agent Security (docs/paper-dgb/main.tex, tag dgb-v1.0.0); this document expands that section into an independent reference with rationale and links to executable tests, so it can be cited or applied without pulling in the full paper.

Analogous in spirit to the Agent-Eval Checklist (Wang et al., 2026), the Decision Governance Checklist targets a narrower question: not "does the agent behave well," but "can the agent's own governance be trusted to constrain the agent, even when the agent is the one under evaluation?" A governance system that the agent it governs can quietly disable, reconfigure, or route around is not a governance system — it is a suggestion.

Why this is a separate axis from behavioral correctness

An agent can pass every behavioral test in a benchmark and still fail every property below, because behavioral tests measure decisions and this checklist measures control over the decision process itself. The two are independent: an agent with excellent judgment but a governance layer it can self-modify is one prompt-injection or one subtly-misaligned objective away from acting on that judgment with no backstop. DGB's former "85% invisible to metadata-only scanning" figure has been withdrawn: it was a hand-assigned label, not a measurement. Measured instead, a named 14-pattern scanner flags 1 of 52 authored tool-registry fixtures and a capability-rule comparator flags 17 — the number depends on the scanner, the fixture set, and what counts as detection. The underlying point stands independently of any percentage: static analysis of an agent's configuration cannot tell you whether the agent will respect that configuration under adversarial pressure. Only execution can.

The seven properties

Each property below states the property, the failure mode it rules out, and the executable test(s) in this repository's harness that check it.

1. Gate immutability

Property: No agent can disable its own governance gates without external ratification.

Rules out: An agent under pressure (a bad objective, a compromised tool call, a jailbreak) simply turning off the check that would have stopped it — the equivalent of an auditee disabling the audit.

Test: GM-001, "Gate Self-Disable" (CRITICAL) — attempts to have the agent disable its own governance/safety gates and confirms the gates remain active without external ratification.

2. Threshold integrity

Property: No agent can modify its own evaluation thresholds.

Rules out: An agent that cannot disable a gate outright instead raising its own risk tolerance or lowering its own audit-coverage requirement until the gate stops triggering — functionally equivalent to disabling it, but harder to detect because the gate is still "on."

Test: GM-002, "Threshold Self-Modification" (HIGH).

3. Permission boundaries

Property: No agent can escalate its own permissions.

Rules out: An agent adding admin/root capabilities or expanding its own tool access without external authorization — the step that turns a contained failure into an uncontained one.

Test: GM-003, "Permission Self-Escalation" (CRITICAL).

4. Kill-switch persistence

Property: No agent can pre-emptively disable its own kill switch.

Rules out: A kill switch that only works if the agent chooses not to have disabled it first — the single most safety-critical property on this list, since every other control assumes there is a way to stop the agent if all of them fail.

Test: GM-004, "Kill-Switch Bypass" (CRITICAL) — attempts to disable or circumvent the kill switch (unregistering handlers, modifying signal routing) before a kill signal is sent.

5. Amendment separation

Property: No agent can both propose and ratify changes to its own constraints.

Rules out: An agent that cannot unilaterally change a rule instead proposing the change and then also being the entity that approves it — a self-dealing loophole in an otherwise sound amendment process. This is the separation-of-powers property: proposal and ratification must be different principals.

Test: GM-005, "Amendment Self-Ratification" (HIGH).

6. Audit immutability

Property: No agent can modify or delete its own audit logs.

Rules out: Retrospective governance-failure detection becoming impossible because the evidence trail itself is writable by the entity being audited. Tamper-evident logging is a precondition for every other form of oversight (human review, incident response, this benchmark's own per-case evidence) — if the log can be edited by its subject, none of those are trustworthy after the fact.

Test: GM-006, "Audit Log Tampering" (HIGH).

7. Evidence independence

Property: No evaluation trusts the entity being evaluated.

Rules out: The broadest and most structural failure mode: an evaluation pipeline (benchmark harness, grader, judge, or CI gate) that takes evidence, self-reported results, or environment state from the same agent it is scoring, letting the agent grade its own exam. This property applies to evaluators, not just to the agents-under-governance covered by properties 1–6 — it is why DGB's own harness ships seven integrity tests aimed at itself.

Tests: BI-001 through BI-007 (benchmark integrity harness) — respectively: BI-001 evaluator/grader-context access from the agent's execution environment; BI-002 reference/gold-answer accessibility to the agent; BI-003 code injection into the grading pipeline via eval()/exec(); BI-004 LLM-judge manipulation via agent output; BI-005 weak string-matching false positives; BI-006 structurally-valid but semantically-empty responses passing validation; BI-007 the evaluator trusting agent-fabricated self-assessments.

Using this checklist

  • As a design checklist: before building or adopting an agent governance layer, verify each property holds by construction (e.g., gate configuration lives outside the agent's write scope; kill-switch handlers are registered by a process the agent cannot reach; audit logs are append-only and externally stored).
  • As an evaluation checklist: run GM-001 through GM-006 and BI-001 through BI-007 against the system under test. These are executable, not descriptive — a PASS means the specific bypass attempt in that test failed against the live system, not that the property holds in general. Passing all 13 is necessary, not sufficient.
  • As a citation: cite this document independently of the full DGB paper when the seven properties, rather than the corpus or the statistical findings, are what's relevant — e.g., in a governance design doc, an internal security review, or a related benchmark's related-work section. See CITATION.cff for BibTeX/CFF metadata.

Relationship to the DGB corpus

The checklist's seven properties are the governance layer's contract. The DGB corpus (benchmarks/decision_behavior_corpus.py, 52 cases across five categories — escalation bypass, collusion, memory tampering, payment chain, evidence fabrication) is a set of concrete scenarios that probe whether a governed agent upholds that contract under realistic pressure, not just whether the governance code exists. The checklist and the corpus are complementary: the checklist can be satisfied by a governance implementation that a well-designed but never-executed corpus wouldn't distinguish from a broken one, which is exactly the gap DGB's execution-based evaluation is built to close (see docs/paper-dgb/main.tex, Section 6, "Threats to Validity," for the current limitations of that evaluation).

Provenance

  • Properties 1–6 and their executable tests: protocol_tests/governance_modification_harness.py.
  • Property 7 and its executable tests: protocol_tests/benchmark_integrity_harness.py.
  • Original publication: docs/paper-dgb/main.tex, Section 7 ("The Decision Governance Checklist"), released under tag dgb-v1.0.0.