Feature Request
Standardize how checks select values from the trace: one naming convention and comparison defaults that match judges (trace.last.outputs when unset).
Motivation
Built-ins mix key, text_key, answer_key, actual_answer_key, reference_text_key. Comparisons require key=; judges often default to trace.last.outputs. Easy to misconfigure.
Implementation plan
- Inventory built-in check constructors and their extraction params (table below → finalize in PR).
- Add default
key="trace.last.outputs" (or shared constant) on comparison checks — additive.
- Document a single reference table in
libs/giskard-checks/README.md.
- Optional aliases: e.g. keep
answer_key as alias of key on judges with DeprecationWarning only if renaming; prefer documenting current names first if defaults alone fix DX.
- Unit tests:
Equals(expected="Paris") without key= reads trace.last.outputs.
Reference table (draft — adjust to match code)
| Check family |
Primary value |
Default path |
Second value |
Comparisons (Equals, …) |
key |
trace.last.outputs |
expected / threshold args |
| String / regex |
text_key (or unify to key) |
trace.last.outputs |
pattern / substr |
| Semantic similarity |
actual_answer_key + reference_text_key |
outputs / metadata.reference_text |
— |
| Judges (Groundedness, …) |
answer_key |
trace.last.outputs |
context / metadata |
API usage (before → after)
from giskard.checks import Scenario, Equals, Groundedness
# Before: comparisons force an explicit key
Scenario("capitals").interact(inputs="...", outputs=get_answer).check(
Equals(key="trace.last.outputs", expected="Paris")
)
# After: same default as judges
Scenario("capitals").interact(inputs="...", outputs=get_answer).check(
Equals(expected="Paris")
)
# Judges unchanged in spirit
Scenario("grounded").interact(inputs="...", outputs=get_answer).check(
Groundedness(context="France is in Western Europe. Capital: Paris.")
)
README snippet shape
### Selecting values from the trace
Most checks read `trace.last.outputs` by default. Override with `key=` / `*_key=`:
| Parameter | Typical default |
| --- | --- |
| `key` / `answer_key` / `text_key` | `trace.last.outputs` |
| context for groundedness | `trace.last.metadata.context` or constructor `context=` |
Acceptance criteria
Out of scope
- Redesigning JSONPath
- Judge prompt text
Feature Request
Standardize how checks select values from the trace: one naming convention and comparison defaults that match judges (
trace.last.outputswhen unset).Motivation
Built-ins mix
key,text_key,answer_key,actual_answer_key,reference_text_key. Comparisons requirekey=; judges often default totrace.last.outputs. Easy to misconfigure.Implementation plan
key="trace.last.outputs"(or shared constant) on comparison checks — additive.libs/giskard-checks/README.md.answer_keyas alias ofkeyon judges withDeprecationWarningonly if renaming; prefer documenting current names first if defaults alone fix DX.Equals(expected="Paris")withoutkey=readstrace.last.outputs.Reference table (draft — adjust to match code)
Equals, …)keytrace.last.outputsexpected/ threshold argstext_key(or unify tokey)trace.last.outputsactual_answer_key+reference_text_keymetadata.reference_textanswer_keytrace.last.outputscontext/ metadataAPI usage (before → after)
README snippet shape
Acceptance criteria
keyusing the judge-aligned defaultOut of scope