Skip to content

feat(checks): unify extraction key parameters and defaults #2659

Description

@davidberenstein1957

Feature Request

Standardize how checks select values from the trace: one naming convention and comparison defaults that match judges (trace.last.outputs when unset).

Motivation

Built-ins mix key, text_key, answer_key, actual_answer_key, reference_text_key. Comparisons require key=; judges often default to trace.last.outputs. Easy to misconfigure.

Implementation plan

  1. Inventory built-in check constructors and their extraction params (table below → finalize in PR).
  2. Add default key="trace.last.outputs" (or shared constant) on comparison checks — additive.
  3. Document a single reference table in libs/giskard-checks/README.md.
  4. Optional aliases: e.g. keep answer_key as alias of key on judges with DeprecationWarning only if renaming; prefer documenting current names first if defaults alone fix DX.
  5. Unit tests: Equals(expected="Paris") without key= reads trace.last.outputs.

Reference table (draft — adjust to match code)

Check family Primary value Default path Second value
Comparisons (Equals, …) key trace.last.outputs expected / threshold args
String / regex text_key (or unify to key) trace.last.outputs pattern / substr
Semantic similarity actual_answer_key + reference_text_key outputs / metadata.reference_text
Judges (Groundedness, …) answer_key trace.last.outputs context / metadata

API usage (before → after)

from giskard.checks import Scenario, Equals, Groundedness

# Before: comparisons force an explicit key
Scenario("capitals").interact(inputs="...", outputs=get_answer).check(
    Equals(key="trace.last.outputs", expected="Paris")
)

# After: same default as judges
Scenario("capitals").interact(inputs="...", outputs=get_answer).check(
    Equals(expected="Paris")
)

# Judges unchanged in spirit
Scenario("grounded").interact(inputs="...", outputs=get_answer).check(
    Groundedness(context="France is in Western Europe. Capital: Paris.")
)

README snippet shape

### Selecting values from the trace

Most checks read `trace.last.outputs` by default. Override with `key=` / `*_key=`:

| Parameter | Typical default |
| --- | --- |
| `key` / `answer_key` / `text_key` | `trace.last.outputs` |
| context for groundedness | `trace.last.metadata.context` or constructor `context=` |

Acceptance criteria

  • Comparisons work without explicit key using the judge-aligned default
  • README table lists extraction parameters for built-ins
  • Aliases preserved if any public names change
  • Unit tests cover default key resolution

Out of scope

  • Redesigning JSONPath
  • Judge prompt text

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions