Shrinking challenges

[nth-commit]

Hey @dubzzz,

Just wondering if you were aware of this github repo: https://github.com/jlink/shrinking-challenge

Would be super interesting to see how fast-check performs in those challenges. Would love to help contribute, but at the moment my time is going into developing a modern pbt library for .NET.

[dubzzz]

Hey @nth-commit,

I have an issue opened regarding that challenge, see #899. I already started to work on it: dubzzz/shrinking-challenge@2630c12 but I am currently working on enhancing my root shrinkers (mostly integer one) as they perform poorly under some circumstances. I think I will join the challenge after that shrinker change (next minor).

For the record, here is an example of really slow shrinking process I recently observed in my CI: #1341 (comment). I have already some ideas* to fix it and extremely increase the performances of shrinkers in general (as they more or less all depends of integer).

*The API ArbitraryWithShrink implemented by integer does not give enough context concerning previous shrinks. It only tell the shrink method that it shrunk once. Ideally it should also gives more context like: the minimal value you've tried was, the maximal one was...

[nth-commit]

Cool, thanks for sharing!

[dubzzz]

@nth-commit I opened a PR to add fast-check into the challenge: jlink/shrinking-challenge#19

[nth-commit]

Awesome! So good to see 😄 Nice work!

[dubzzz]

Lots of work to come to be better than the current state

[nth-commit]

Great to have it measured.

Honestly, I'm not sure all of those shrinking situations are super useful to chase, with 100% normalization. I reckon each one needs to be weighed up against the complexity it might add to the library, which you'll probably have a pretty good intuition about.

It's great that they exist anyway, purely as a measurement/comparison/aspirational set of situations that crosses platforms. I have a few other challenges I've thought of that I may add at some point.

KUTGW 🙌

[dubzzz]

fast-check is now part of the challenge 🎉

See https://github.com/jlink/shrinking-challenge

[kinthaiofficial]

Shrinking in property-based testing maps interestingly onto multi-agent systems — the "minimal failing case" problem is exactly what you want when debugging a delegation chain gone wrong.

For agent system testing, property-based testing with shrinking is valuable for:

Protocol invariants — "for any sequence of delegation operations, the child's capability set is always a subset of the parent's" — properties that should hold across all possible agent interaction sequences. Fast-check can generate arbitrary agent command sequences and find the minimal one that breaks the invariant.

Budget invariants — "for any sequence of agent spawns and task executions, the total cost never exceeds the root budget" — shrinking finds the minimal spawning pattern that causes a budget overrun, which is usually more useful for debugging than the full complex sequence.

State machine coverage — agent execution is a state machine (pending → running → delegated → budget-paused → completed/failed). Property tests can verify that no sequence of inputs can leave the agent in an undefined state.

The specific challenge with shrinking in agent tests is that agent state is stateful — the minimal failing sequence isn't just a minimal input, it's a minimal sequence of actions. Fast-check's stateful model-based testing is the right primitive for this.

We've been applying property-based testing to KinthAI's agent protocol layer — the budget invariants and capability narrowing are natural candidates: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons covers the protocol design that we test.

Is the shrinking challenge primarily for complex data structures, or for long command sequences?