Ideas from DSPy: self-improving agents, composable reasoning modules, and LLM call caching

[threepointone]

Context

Deep analysis of DSPy (Stanford NLP's "programming — not prompting — language models" framework, 33k stars, v3.1.3) to identify ideas worth borrowing or adapting for the Agents SDK. DSPy's core thesis — prompts should be derived artifacts optimized by algorithms, not hand-written strings — maps surprisingly well onto a durable, stateful runtime like ours.

---

1. Built-in LLM Call Caching in Durable Storage

DSPy pattern: Every LLM call is cached to disk by default (diskcache + xxhash for cache keys). This makes eval loops reproducible and avoids redundant API costs.

How it maps to us: We already have SQLite via Durable Objects (this.sql). A transparent cache that hashes model + messages + tools + temperature and stores responses in a cf_agents_llm_cache table would:

• Eliminate redundant API calls across retries, fibers, and re-evaluations
• Enable the eval/optimization ideas below (they depend on cheap replays)
• Track hit rates via existing observability channels

Effort: Low — the storage layer already exists.

---

2. Composable Reasoning Modules

DSPy pattern: Pre-built modules (ChainOfThought, ReAct, BestOfN, Refine, MultiChainComparison) that compose like building blocks. Users mix and match instead of reimplementing common patterns.

How it maps to us: The guides/anthropic-patterns/ directory already demonstrates sequential chains, routing, parallel execution, orchestrator, and evaluator patterns — but as one-off guide code. These could be extracted into reusable abstractions:

• BestOfN — Run N completions, pick the best by a metric (fibers via withFibers already support parallel execution)
• Refine — Self-critique loop: generate → evaluate → regenerate if score is low (Session.onCompaction() is already this shape)
• ChainOfThought — Automatically inject reasoning fields (the ReasoningUIPart in AIChatAgent already supports this concept)
• Consensus — Run with multiple models, take majority vote

These would compose with onChatMessage in AIChatAgent.

Effort: Medium.

---

3. Evaluation / Metrics as Runtime Primitives

DSPy pattern: Evaluate is a first-class citizen. You define a metric, run your program over a dataset, and get a score. Optimizers then search for configurations that maximize that score.

How it maps to us: We have evals/ with evalite, but only as a CI artifact. The idea is making eval a runtime-accessible primitive. Since every Agent is a Durable Object with its own SQLite, it could:

• Log every LLM interaction + outcome
• Accumulate its own performance history
• Expose metrics via the existing observability channels (adding a metrics channel alongside state, rpc, message, etc.)

This turns each agent instance into a self-monitoring system.

Effort: Medium.

---

4. Automatic Context Window Management

DSPy pattern: Optimizers manage what goes into the context window — which few-shot examples, which instructions, how much. BootstrapFewShot selects the best demonstrations from a pool. KNNFewShot retrieves nearest-neighbor examples at inference time.

How it maps to us: The experimental Session class already has the pieces:

• withContext("soul", { maxTokens: 1100 }) — context blocks with token budgets
• onCompaction() — message history compression
• truncateOlderMessages() — read-time context truncation

What's missing is automatic selection. Instead of the user manually choosing what context to include, the SDK could:

• Maintain a pool of successful interaction examples in SQLite
• Automatically select the most relevant examples per query
• Use observability data to learn which context configurations produce the best outcomes

This turns Session from a passive storage layer into an active context optimization engine.

Effort: Medium-High.

---

5. Durable Prompt Versioning and A/B Testing

DSPy pattern: Optimizers produce versioned, serializable "compiled" programs. You can save, load, and compare configurations (dspy.load("optimized_v3.json")).

How it maps to us: Since every Agent has its own SQLite:

• Track which system prompt / context config / tool set was used per interaction
• Measure outcomes per configuration
• Support A/B testing natively — different agent instances (by name) run different configs
• Roll back to previous configurations if metrics degrade

Session.freezeSystemPrompt() / Session.refreshSystemPrompt() already handle prompt snapshots. Adding version tracking + metric correlation would turn this into live prompt optimization.

Effort: Medium.

---

6. Declarative Task Signatures

DSPy pattern: You declare what a module does ("question -> answer") as a typed signature, not how to prompt for it. The framework handles prompt construction, output parsing, and retry logic.

How it maps to us: Users of AIChatAgent manually construct system prompts, tool descriptions, and wire everything together. A declarative layer where you define capabilities as typed I/O contracts could auto-generate system prompts, tool descriptions, and response parsing:

class MyAgent extends AIChatAgent {
  @task("question: string -> answer: string")
  async answerQuestion(question: string) { /* ... */ }
}

Effort: High — requires a new abstraction layer.

---

7. Self-Improving Agents (The Big One)

The synthesis of DSPy's optimization philosophy with our durable runtime.

DSPy runs optimization offline and deploys the result. Because our agents are durable — they persist state across requests, have SQLite, survive evictions — they can optimize themselves at runtime:

1. Log every LLM interaction + outcome in durable SQLite (see Version Packages #3 above)
2. Periodically (via this.schedule()) run a self-improvement step:
   • Analyze which system prompt variants produced the best outcomes
   • Select the best few-shot examples from recent successful interactions
   • Adjust tool descriptions based on tool-call success rates
3. Apply the improved configuration for subsequent requests

No offline pipeline required. Each agent instance literally gets better over time.

This is something no other framework can do because nobody else has per-agent persistent state at this level. DSPy proves the optimization algorithms work; our runtime provides the durable substrate to run them continuously.

Effort: High — builds on all the above.

---

Suggested Priority

Idea	Effort	Impact	Builds on
LLM call caching in SQLite	Low	High	this.sql, Session
Composable reasoning modules (BestOfN, Refine)	Medium	High	Fibers, AIChatAgent
Eval/metrics as runtime primitives	Medium	High	Observability channels
Automatic context selection	Medium-High	High	Session, context blocks
Durable prompt versioning + A/B	Medium	Medium	Session.freezeSystemPrompt
Declarative task signatures	High	High	New abstraction
Self-improving agents	High	Very High	All of the above

References

• DSPy repo
• DSPy paper (ICLR 2024)
• GEPA: Reflective Prompt Evolution (Jul 2025)
• DSPy docs

[threepointone]

Addendum: DSPy's RLM (Recursive Language Model) — and why our runtime can do it better

After filing this issue, came across dspy/predict/rlm.py which deserves separate analysis. It's DSPy's most ambitious module and maps directly onto infrastructure we already have.

What RLM does

RLM flips the standard approach to long-context processing. Instead of stuffing a massive document into the context window, RLM gives the LLM a sandboxed Python REPL and lets it write code to programmatically explore the data. The LLM becomes a programmer investigating its inputs rather than a reader processing them in one pass.

The execution loop:

1. LLM sees metadata about variables (not the full content) + a REPL history of prior iterations
2. LLM writes Python code to explore/process the data (print samples, check types, slice strings, regex search, etc.)
3. Code executes in a sandboxed Pyodide/WASM interpreter, stdout is captured
4. LLM sees the output, decides what to do next
5. Repeat until the LLM calls SUBMIT() with final outputs, or max_iterations is reached
6. If max iterations hit without SUBMIT(), a fallback extract module reads the full trajectory and forces a final answer

The critical feature: the LLM has access to llm_query(prompt) and llm_query_batched(prompts) — it can recursively call sub-LLMs on slices of the data. So for a 500K character document, instead of feeding it all into one context window, the LLM might write:

# Inspect the data first
print(f"Document length: {len(context)} chars")
print(f"First 500 chars: {context[:500]}")

Then in the next iteration:

# Split into chunks and analyze in parallel
chunks = [context[i:i+5000] for i in range(0, len(context), 5000)]
summaries = llm_query_batched([f"Extract key facts from:\n{c}" for c in chunks])
for i, s in enumerate(summaries):
    print(f"Chunk {i}: {s[:200]}")

Then:

# Synthesize from summaries
combined = "\n".join(summaries)
answer = llm_query(f"Given these facts:\n{combined}\n\nAnswer: {query}")
SUBMIT(answer=answer)

The LLM chooses its own decomposition strategy — map-reduce, tree search, targeted regex, whatever fits the problem. The framework doesn't prescribe the approach.

What's clever about the design

1. Resource budgeting as a prompt constraint. The LLM is told "You have max N sub-LLM calls" and must allocate them wisely. The instructions say "Use Python code for aggregation instead of making more LLM calls." This creates economic pressure toward efficient solutions — the LLM learns to do string processing in code and reserve LLM calls for semantic understanding.

2. Batched concurrent sub-LLM calls. llm_query_batched runs prompts concurrently via ThreadPoolExecutor. The LLM can process N chunks in parallel without the framework needing to know about parallelism upfront. The LLM discovers parallelism opportunities itself.

3. Persistent REPL state across iterations. Variables from one code block carry over to the next. The LLM builds up intermediate results across multiple steps — this is iterative reasoning with working memory, not a single-shot prompt.

4. The "explore first" instruction pattern. The action prompt explicitly says: "EXPLORE FIRST — Look at your data before processing it. Print samples, check types/lengths, understand the structure." This is a data-science workflow encoded as an LLM instruction — inspect before you process.

5. Pluggable interpreter. The CodeInterpreter is an interface. Default is Pyodide/WASM, but you can swap in E2B, Modal, or a mock for testing. User-provided tools are normalized and injected alongside the built-in llm_query/llm_query_batched.

6. Typed output validation. When the LLM calls SUBMIT(), outputs are validated against the signature's Pydantic types. Type mismatches produce error messages fed back to the LLM so it can fix and retry.

What's less clever

Ephemeral execution. RLM runs entirely in-process. If the process dies mid-iteration, all progress is lost. There's no checkpointing, no recovery from crashes, no persistence of the trajectory for later replay.

Thread-based concurrency. llm_query_batched uses Python's ThreadPoolExecutor with a default of 8 workers. The class docs explicitly warn "RLM instances are not thread-safe when using a custom interpreter." This is fine for notebooks, not for serving.

The action prompt is hand-written. There's literally a # TODO: Optimize this prompt across a diverse benchmark at the top of the file. Ironic for a framework whose entire thesis is "don't hand-write prompts" — DSPy's own optimizers haven't been applied to RLM's core prompt.

No cost tracking or observability. The call_count state tracks how many sub-LLM calls have been made (for budget enforcement), but there's no tracking of tokens consumed, latency per iteration, or cost. In production you'd want to know "this RLM execution used 150K input tokens across 12 sub-LLM calls over 8 iterations."

How this maps to our existing infrastructure

This is where it gets interesting. We already have the building blocks:

RLM concept	Agents SDK equivalent
PythonInterpreter (Pyodide/WASM sandbox)	DynamicWorkerExecutor from @cloudflare/codemode (isolated V8 Worker)
llm_query() / llm_query_batched()	Workers AI calls from inside the executor, or tools exposed via stateTools
REPL state persisting across iterations	stashFiber() checkpointing in SQLite via withFibers
max_iterations loop	Fiber retry loop with maxRetries
SUBMIT() for final output	Fiber completion with onFiberComplete()
max_llm_calls budget	Natural in a billing-aware runtime (could track via observability channels)
Trajectory for debugging	Already stored in SQLite if built on fibers
User-provided tools	MCP tools via MCPClientManager, or Tool from the AI SDK

What a durable RLM would look like on our runtime

An RLM-style module built on Durable Objects + fibers would improve on DSPy's version in several ways:

1. Crash-resilient execution. Each iteration's REPL history would be persisted in SQLite via stashFiber(). If the Durable Object is evicted mid-iteration, the fiber recovery mechanism (onFiberRecovered) restarts from the last checkpoint instead of losing all progress. DSPy's RLM loses everything on crash.

2. True parallel sub-LLM calls. Instead of Python threads, sub-LLM calls could be separate fibers or service bindings to Workers AI. These run in V8 isolates with real concurrency, not GIL-constrained threads. Batched calls could fan out to multiple Workers.

3. Observable execution. Every iteration could emit events on the existing observability channels (agents:message for code execution, agents:rpc for sub-LLM calls). The trajectory would be visible in Tail Workers, not just as a return value.

4. Cost-aware budgeting. The track_usage pattern (which DSPy has separately, but doesn't integrate into RLM) could be built into the iteration loop. Each sub-LLM call would record tokens consumed, and the budget constraint could be expressed in tokens or dollars rather than just call count.

5. Cached sub-LLM calls. If we implement the LLM call caching from idea #1 in this issue, repeated sub-LLM calls with identical prompts (common when retrying or when multiple chunks produce similar sub-queries) would hit the cache instead of the API.

6. Persistent trajectory for replay and optimization. The full trajectory (code written, outputs observed, sub-LLM calls made, final result) stored in SQLite becomes training data. Over time, an agent could learn which decomposition strategies work best for different input types — connecting to the self-improving agents idea (#7 in this issue).

Concrete next step

The lowest-friction way to prototype this: build an example that uses @cloudflare/codemode's DynamicWorkerExecutor in an iterative loop inside an AIChatAgent, where:

• The LLM generates JavaScript/TypeScript (not Python) to run in the executor
• The executor has access to a query() function that calls Workers AI
• Each iteration's code + output is stored in the Session history
• The loop runs inside a fiber so it survives eviction

This would be a "Workers-native RLM" — same pattern, durable execution, no Python dependency.

Reference

• DSPy RLM source
• Recursive Language Models paper (Zhang, Kraska, Khattab, 2025)
• Related internal: @cloudflare/codemode, @cloudflare/shell, withFibers in packages/agents/src/experimental/forever.ts

[MatsErdkamp]

I think the slop writeup is actually pretty okay..!

I think there are really three pieces to think about here: signatures, optimizers, and modules.

Signatures are very powerful, though I could imagine them living outside this package rather than being treated as core to it. (https://dspy.ai/learn/programming/signatures/)

Optimizers are harder to separate, because they more or less require signatures to exist in the first place. If optimization is in scope, signatures probably are too. GEPA feels especially relevant here. (https://github.com/gepa-ai/gepa)

Then there are modules, especially reasoning language models (RLMs), which seem particularly compelling for Cloudflare Agents. Recursive REPL-style environments map nicely onto dynamic workers, so this feels like a genuinely natural fit for the platform rather than just a port of DSPy concepts.

All that said, I do think signatures and optimization are both extremely powerful and still underexplored in JavaScript. Personally I’d make them first-class citizens as well, even if some people would see that as adding too much surface area.

Here’s an example use-case, just to make this more concrete.

I have a bunch of product exports from various partners in an R2 bucket, and I want to unify them into one product catalog for my webshop. The files are too large to sensibly stuff into a single context window, and each partner uses a different structure / naming scheme / taxonomy.

A signature is useful here because I can define the target transformation at a high level:

partner_export -> normalized_product[]

or more explicitly:

raw row/group -> { title, description, brand, price, images, attributes, category }

That gives me a consistent contract independent of how messy the source data is.

A module is then the thing that actually executes that process. In the simple case that might just be extraction + normalization. In the more interesting case it might be an RLM that inspects the file, samples rows, notices that variants are split across tabs, writes some parsing logic, runs it, inspects the output, and iterates until it has a clean normalized result.

That is where Cloudflare Agents feels particularly interesting to me: the recursive REPL loop maps well to dynamic workers. Instead of trying to cram the whole export into one prompt, the agent can spin up the right tools/processes to inspect chunks, recurse into weird edge cases, and gradually build the normalized catalog.

Then optimizers become useful once you want to improve that pipeline over time. For example, maybe I have a small set of known-good normalized catalogs, or some downstream metrics like category accuracy, variant grouping accuracy, or attribute coverage. An optimizer could tune the prompts, decomposition strategy against those metrics.

This is where GEPA is especially interesting to me, since it can use Actionable Side Information (ASI) as part of the optimization loop. In other words, it can reflect on evaluator feedback and run-time diagnostics, then rewrite the artifact accordingly rather than optimizing blindly. The newer optimize_anything API seems especially relevant here. (https://gepa-ai.github.io/gepa/blog/2026/02/18/introducing-optimize-anything/)

So to me the interesting part is that these three layers compose nicely:

• signatures define the contract
• modules / RLMs do the work
• optimizers improve the system over time

[MatsErdkamp]

Another way to look at this more abstractly is to make the durable agents self-evolving and possibly use-case-specific.

Say we have a bugbot agent. The signature is just code -> feedback. It starts off quite generic.

If we then use it for a while, let it observe which feedback gets accepted vs disregarded, and maybe even scan feedback from human reviewers on the PR, we start to make that agent much more specific to the actual engineering culture and review norms of a given codebase.

At that point the agent is no longer just “a generic code review bot.” It becomes a reviewer that gradually adapts to the kinds of issues this team actually cares about: correctness bugs, performance regressions, security footguns, style violations, missing tests, API consistency, etc.

That is where the signatures / modules / optimizers split becomes interesting again.

The signature can stay stable: code -> feedback.

The module defines how the review is actually carried out. Maybe it just reads the diff and comments. Maybe it also pulls surrounding files, runs lightweight checks, inspects test changes, or recursively explores suspicious callsites.

Then the optimizer is what makes the agent improve over time. It can look at which comments were accepted, which were ignored, which human comments appeared later, and use that signal to refine the review strategy or prompt. Over time, the bot starts to sound less like a generic LLM and more like a reviewer that is actually calibrated to the repository.

To me that feels like a very natural fit for durable agents: not just long-lived state, but long-lived adaptation.

And in the Cloudflare Agents world, that seems especially compelling, because the runtime already gives you a place for these agents to persist state, keep traces/logs, and evolve based on repeated interaction with a specific environment.

Let me know if this makes sense!

[kinthaiofficial]

Excellent analysis — DSPy's thesis that prompts should be derived artifacts is genuinely profound, and mapping it onto a durable stateful runtime raises some fascinating architectural questions.

From operating 31 agents in production on OpenClaw, I'd add a few patterns that complement the DSPy ideas:

On LLM call caching in durable storage: The tricky part is cache invalidation when the agent's memory state changes. We found that caching at the (prompt_template_hash, memory_snapshot_hash, tool_state) tuple level works better than raw prompt caching. This avoids stale responses when the agent has learned something new since the last identical-looking call.

On composable reasoning modules: DSPy's ChainOfThought/ReAct composability maps well to what we call per-skill model routing — each reasoning module can target a different model based on cost/latency/capability requirements. A planning module routes to a stronger model, a simple extraction step routes to a cheaper one. The Agents SDK already has the durable execution foundation; adding a model_selector at the module level would be a natural extension.

On self-improving agents: The most practical version we've deployed is Trace2Skill promotion — the system analyzes successful execution traces, identifies recurring patterns, and promotes them to first-class skills with proper error handling. Combined with DSPy-style optimization, you could automatically tune the promoted skill's prompts against a holdout set of similar tasks.

One caveat on the "dreaming" / memory consolidation idea: when agents consolidate memories offline, you need both deduction and induction specialists — deduction prunes contradictions, induction identifies emerging patterns. Without both, consolidation either loses important signals or accumulates noise.

Related architecture work if useful:

• Persistent memory architecture (importance scoring, time decay, compaction): https://blog.kinthai.ai/why-character-ai-forgets-you-persistent-memory-architecture
• Multi-agent coordination at scale (221 agents, delegation chains): https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons

[kinthaiofficial]

Great analysis. DSPy's 'derived artifacts, not handwritten prompts' thesis maps well to durable agent runtimes. Some additional patterns we've found valuable from running 31 agents in production:

LLM Call Caching: content-addressable with model version pinning

DSPy's diskcache approach works for single-agent eval loops but breaks in multi-agent settings where different agents might need cache isolation. We use content-addressable caching with a composite key:

cacheKey = hash(modelId + modelVersion + canonicalize(messages) + temperature + toolManifest)

The toolManifest is important — the same prompt with different available tools produces different outputs. Miss this and you get cache poisoning across agent types.

Self-improving agents: Trace2Skill pattern

DSPy's optimizer iterates over prompts. We go further — we promote successful execution traces into reusable skills:

1. Agent completes a novel task successfully
2. System extracts the action sequence as a candidate skill
3. Deduction specialist validates the logic; induction specialist generalizes across similar traces
4. If the pattern appears 3+ times with >80% success rate, it's promoted to a named skill

This is essentially DSPy's optimization but at the behavioral level rather than prompt level. The agent's capability set grows over time without explicit programming.

Composable reasoning: progressive context compaction

DSPy's modules compose because they have clean interfaces. For agents with long-running conversations, we use three-tier context compaction:

• Full (recent): verbatim conversation
• Structured summary (medium): key entities, decisions, and open questions preserved
• One-line digest (old): compressed to a single reference line

Critically, entity references are preserved verbatim through compaction — if an agent discussed a specific API endpoint or variable name, that exact string survives even in the digest tier. This is the provenance tracking that makes composed reasoning modules reliable across context boundaries.

Related reading:

• Why Character.AI Forgets You — Persistent Memory Architecture — the memory architecture that makes self-improvement persistent
• OpenClaw Multi-Tenancy: Why VM-per-User Doesn't Scale — caching strategy at the infrastructure level

Running this at agents.kinthai.ai — the Trace2Skill pattern has been the biggest win for agent capability growth.