docs(oss): add OpenAI Agents SDK/MCP and reliable-judge tutorials - #184
docs(oss): add OpenAI Agents SDK/MCP and reliable-judge tutorials#184davidberenstein1957 wants to merge 4 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
giskard-docs | 030a701 | Commit Preview URL Branch Preview URL |
Aug 13 2026, 06:11 AM |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Out of scope for this cycle, let's postpone the review of this PR |
|
Hi @henchaves why is this out of scope? I thjough we had discussed we wanted to include documentation updates, more tailored tutorials and a general documentation cleanup? |
|
advanced checks tutorials are not priority at the moment, first we need to fix checks API reference and add scan pages. We can add more once we have a stable release candidate, since we can still break things until there |
|
Alright, then I misunderstood. I've also added some PRs related to the API reference updates and more advanced scan documentation stucture according to diataxis. Feel free to take a look at those first. |
|
It's ok, I left these ones you mentioned as open to review |
Problem
Two second-wave gaps. (a) After the LangGraph tutorial there is exactly one framework integration; the OpenAI Agents SDK plus MCP tool servers is the other mainstream stack and had no page. (b)
RegoPolicyand the semantic/judge checks are documented as isolated reference entries andexplanation/when-to-use-which-check.mdargues the choice abstractly — nothing walks a reader from a flaky LLM judge to a deterministic check, which is the most common real complaint about LLM evaluation.Changes
checks/tutorials/test-openai-agents.ipynb: an Agents SDK agent with an MCP server,Runner.run(...)wrapped as the async Giskard target, session state via aTracesubclass; asserts the final answer, that the MCP tool was actually invoked (WithSpy), and behaviour when the MCP server errors or returns empty. Callout that this is the same target contract as the LangGraph tutorial.checks/tutorials/reliable-judges.ipynb: start from a naive judge on a subjective requirement, run it N times over the same trace and measure the disagreement rate, then re-express the same requirement as a deterministicFnCheck, a non-trivialRegoPolicy, a narrowed semantic check, and finally a tightened-rubric judge with a pinned model. Ends with a variance-based CI threshold recipe linkinghow-to/ci-cd.checks/tutorials/index.mdx.Verification
gpt-4o-mini; the flakiness number in the judge tutorial is produced by the notebook, not asserted in prose..mdxgenerated viascripts/convert-notebooks.mjs;npm run buildpasses.Notes / risks
🤖 Generated with Claude Code