Skip to content

docs(oss): add OpenAI Agents SDK/MCP and reliable-judge tutorials - #184

Draft
davidberenstein1957 wants to merge 4 commits into
mainfrom
docs-plan/item-13-mcp-and-judge-tutorials
Draft

docs(oss): add OpenAI Agents SDK/MCP and reliable-judge tutorials#184
davidberenstein1957 wants to merge 4 commits into
mainfrom
docs-plan/item-13-mcp-and-judge-tutorials

Conversation

@davidberenstein1957

Copy link
Copy Markdown
Member

Problem

Two second-wave gaps. (a) After the LangGraph tutorial there is exactly one framework integration; the OpenAI Agents SDK plus MCP tool servers is the other mainstream stack and had no page. (b) RegoPolicy and the semantic/judge checks are documented as isolated reference entries and explanation/when-to-use-which-check.md argues the choice abstractly — nothing walks a reader from a flaky LLM judge to a deterministic check, which is the most common real complaint about LLM evaluation.

Changes

  • New checks/tutorials/test-openai-agents.ipynb: an Agents SDK agent with an MCP server, Runner.run(...) wrapped as the async Giskard target, session state via a Trace subclass; asserts the final answer, that the MCP tool was actually invoked (WithSpy), and behaviour when the MCP server errors or returns empty. Callout that this is the same target contract as the LangGraph tutorial.
  • New checks/tutorials/reliable-judges.ipynb: start from a naive judge on a subjective requirement, run it N times over the same trace and measure the disagreement rate, then re-express the same requirement as a deterministic FnCheck, a non-trivial RegoPolicy, a narrowed semantic check, and finally a tightened-rubric judge with a pinned model. Ends with a variance-based CI threshold recipe linking how-to/ci-cd.
  • Both registered in checks/tutorials/index.mdx.

Verification

  • Both notebooks executed with outputs committed, model gpt-4o-mini; the flakiness number in the judge tutorial is produced by the notebook, not asserted in prose. .mdx generated via scripts/convert-notebooks.mjs; npm run build passes.

Notes / risks

  • Depends on item 08's target-wrapping and spy patterns — land 08 first.
  • Adds tutorial sidebar entries; check ordering against items 06/09/10, which contend for order 6.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 13, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
giskard-docs 030a701 Commit Preview URL

Branch Preview URL
Aug 13 2026, 06:11 AM

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 13, 2026 06:13
@henchaves

Copy link
Copy Markdown
Member

Out of scope for this cycle, let's postpone the review of this PR

@henchaves
henchaves marked this pull request as draft August 13, 2026 06:30
@davidberenstein1957

Copy link
Copy Markdown
Member Author

Hi @henchaves why is this out of scope? I thjough we had discussed we wanted to include documentation updates, more tailored tutorials and a general documentation cleanup?

Copy link
Copy Markdown
Member

advanced checks tutorials are not priority at the moment, first we need to fix checks API reference and add scan pages. We can add more once we have a stable release candidate, since we can still break things until there

@davidberenstein1957

Copy link
Copy Markdown
Member Author

Alright, then I misunderstood. I've also added some PRs related to the API reference updates and more advanced scan documentation stucture according to diataxis. Feel free to take a look at those first.

Copy link
Copy Markdown
Member

It's ok, I left these ones you mentioned as open to review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants