Design discussion: per-method LLM selection (`method_llms`, `llm_roles`, `Role`)

[mempko]

Motivation

NOOA resolves one LLM per agent (instance → class → parent cascade). In practice, methods within a single agent often want different models: a cheap/fast model for summarize, a strong model for plan, a local model for privacy-sensitive methods. Today the only options are splitting the agent apart or using @strategy(llm=...) — which exists but is undocumented, untested, and requires constructing a concrete UnifiedLLM client at import time, which cuts against the framework's layered-config philosophy (YAML registry, env overrides).

Per CONTRIBUTING.md's guidance to discuss substantial features before a large PR, I'd like feedback on the following design. I have a working implementation with tests (linked below).

Proposed design

Three composable mechanisms, from most local to most configurable:

1. @strategy(llm=...) widened — accepts a UnifiedLLM client, a registry alias / litellm model string (resolved lazily on first call, cached per agent instance), or a Role reference:

class SupportAgent(Agent, llm="gpt-5"):
    @strategy(llm="gpt-5-mini")
    async def summarize(self, text: str) -> str:
        """One-sentence summary."""
        ...

2. method_llms={...} — class/instance kwarg mapping generation-method names to models, so callers (subclassers, instantiators, eval harnesses) can re-route a method without redefining it. MRO-merged, instance over class; deliberately beats @strategy(llm=...) so configuration overrides code-level defaults:

agent = SupportAgent(method_llms={"summarize": experimental_llm})

3. llm_roles={...} + Role("name") — named LLM slots. Methods declare what kind of model they want; classes/instances/hosts decide what that means:

class SupportAgent(Agent, llm="gpt-5", llm_roles={"cheap": "gpt-5-mini"}):
    @strategy(llm=Role("cheap"))
    async def summarize(self, text: str) -> str: ...

agent = SupportAgent(llm_roles={"cheap": local_llm})  # instance binding merges
                                                      # over the class binding

Like method_llms, role bindings layer: MRO-merged class bindings, then instance bindings passed to __init__. A class may even reference a role it doesn't bind, as long as the instance (or a subclass) binds it by construction time.

Runtime reconfiguration — both override mechanisms are mutable on a live agent, taking effect on the next call, so hosts (a TUI model switcher, an eval harness, a cost controller) can re-route methods without reconstructing the agent:

agent.set_llm_role("cheap", local_llm)          # rebind a role — every method
                                                # referencing Role("cheap") follows,
                                                # since roles dereference at call time
agent.set_method_llm("summarize", "gpt-5-mini") # override one method by name
agent.set_method_llm("summarize", None)         # clear it — falls back to
                                                # @strategy(llm=...) / agent default

Both setters accept a client, a registry alias / model string, or (for set_method_llm) a bound Role. A one-off override is also possible per call: await agent.summarize(text, llm=other).

Resolution precedence (highest wins):

call-site llm=  >  instance method_llms  >  class method_llms  >  @strategy(llm=...)  >  agent default

Fail-fast validation, matching the framework's eager-at-__init__, lazy-I/O pattern: unknown method_llms keys, keys naming non-generation methods, and unbound Role references raise at class creation / __init__ (with the list of available roles), not mid-run. String aliases resolve at __init__ through the existing get_llm_client() path.

Host accessors: agent.llm_for(method_name) (resolves without calling), agent.llm_roles snapshot property, and the set_method_llm() / set_llm_role() setters above; set_llm() continues to replace only the default.

No changes were needed to token budgeting, summarization, or tracing — they already key off the per-call _current_llm_var, and TokenCalibration is already per-model.

Bugs found along the way

Two latent issues in the existing (undocumented) call-site path, fixed in the implementation:

• The call-site llm= kwarg handled by _execute_with_generation was unreachable: the method wrapper's ArgumentValidator rejected the kwarg before dispatch. It's now stripped pre-validation, like _session_locals.
• The framework unconditionally popped kwargs["llm"], silently consuming a user parameter named llm. It's now only treated as a framework kwarg when the method doesn't declare its own llm parameter.

Implementation

Working implementation with 38 new tests (all five precedence levels, runtime rebinding by method name and by role, alias caching, standalone functions, validation errors — plus first-time coverage of the pre-existing @strategy(llm=...) path), a quickstart example, and docs: mempko#1. Full suite passes; happy to open it as a PR here if the design direction looks right.

Open questions

1. Precedence: is method_llms beating @strategy(llm=...) the right call? My reasoning: configuration should override code-level defaults, mirroring the llm-config chain and truncation merge semantics.
2. Naming: method_llms / llm_roles / Role — happy to rename to fit project conventions.
3. Scope: Role is intentionally rejected as the agent default llm= (the default resolves eagerly at __init__ and feeds summarizer/window budgeting; making it call-time-lazy seemed too invasive for v1). Reasonable?

[rdasilveiracabral]

@sklinglernv i think you had a proposal to get rid of unifiedllm entirely, so will let you comment here :)

[aleksola]

Motivation

NOOA resolves one LLM per agent (instance → class → parent cascade). In practice, methods within a single agent often want different models: a cheap/fast model for summarize, a strong model for plan, a local model for privacy-sensitive methods. Today the only options are splitting the agent apart or using @strategy(llm=...) — which exists but is undocumented, untested, and requires constructing a concrete UnifiedLLM client at import time, which cuts against the framework's layered-config philosophy (YAML registry, env overrides).

Per CONTRIBUTING.md's guidance to discuss substantial features before a large PR, I'd like feedback on the following design. I have a working implementation with tests (linked below).

Proposed design

Three composable mechanisms, from most local to most configurable:

1. @strategy(llm=...) widened — accepts a UnifiedLLM client, a registry alias / litellm model string (resolved lazily on first call, cached per agent instance), or a Role reference:

class SupportAgent(Agent, llm="gpt-5"):
    @strategy(llm="gpt-5-mini")
    async def summarize(self, text: str) -> str:
        """One-sentence summary."""
        ...

2. method_llms={...} — class/instance kwarg mapping generation-method names to models, so callers (subclassers, instantiators, eval harnesses) can re-route a method without redefining it. MRO-merged, instance over class; deliberately beats @strategy(llm=...) so configuration overrides code-level defaults:

agent = SupportAgent(method_llms={"summarize": experimental_llm})

3. llm_roles={...} + Role("name") — named LLM slots. Methods declare what kind of model they want; classes/instances/hosts decide what that means:

class SupportAgent(Agent, llm="gpt-5", llm_roles={"cheap": "gpt-5-mini"}):
    @strategy(llm=Role("cheap"))
    async def summarize(self, text: str) -> str: ...

agent = SupportAgent(llm_roles={"cheap": local_llm})  # instance binding merges
                                                      # over the class binding

Like method_llms, role bindings layer: MRO-merged class bindings, then instance bindings passed to __init__. A class may even reference a role it doesn't bind, as long as the instance (or a subclass) binds it by construction time.

Runtime reconfiguration — both override mechanisms are mutable on a live agent, taking effect on the next call, so hosts (a TUI model switcher, an eval harness, a cost controller) can re-route methods without reconstructing the agent:

agent.set_llm_role("cheap", local_llm)          # rebind a role — every method
                                                # referencing Role("cheap") follows,
                                                # since roles dereference at call time
agent.set_method_llm("summarize", "gpt-5-mini") # override one method by name
agent.set_method_llm("summarize", None)         # clear it — falls back to
                                                # @strategy(llm=...) / agent default

Both setters accept a client, a registry alias / model string, or (for set_method_llm) a bound Role. A one-off override is also possible per call: await agent.summarize(text, llm=other).

Resolution precedence (highest wins):

call-site llm=  >  instance method_llms  >  class method_llms  >  @strategy(llm=...)  >  agent default

Fail-fast validation, matching the framework's eager-at-__init__, lazy-I/O pattern: unknown method_llms keys, keys naming non-generation methods, and unbound Role references raise at class creation / __init__ (with the list of available roles), not mid-run. String aliases resolve at __init__ through the existing get_llm_client() path.

Host accessors: agent.llm_for(method_name) (resolves without calling), agent.llm_roles snapshot property, and the set_method_llm() / set_llm_role() setters above; set_llm() continues to replace only the default.

No changes were needed to token budgeting, summarization, or tracing — they already key off the per-call _current_llm_var, and TokenCalibration is already per-model.

Bugs found along the way

Two latent issues in the existing (undocumented) call-site path, fixed in the implementation:

• The call-site llm= kwarg handled by _execute_with_generation was unreachable: the method wrapper's ArgumentValidator rejected the kwarg before dispatch. It's now stripped pre-validation, like _session_locals.
• The framework unconditionally popped kwargs["llm"], silently consuming a user parameter named llm. It's now only treated as a framework kwarg when the method doesn't declare its own llm parameter.

Implementation

Working implementation with 38 new tests (all five precedence levels, runtime rebinding by method name and by role, alias caching, standalone functions, validation errors — plus first-time coverage of the pre-existing @strategy(llm=...) path), a quickstart example, and docs: mempko#1. Full suite passes; happy to open it as a PR here if the design direction looks right.

Open questions

1. Precedence: is method_llms beating @strategy(llm=...) the right call? My reasoning: configuration should override code-level defaults, mirroring the llm-config chain and truncation merge semantics.
2. Naming: method_llms / llm_roles / Role — happy to rename to fit project conventions.
3. Scope: Role is intentionally rejected as the agent default llm= (the default resolves eagerly at __init__ and feeds summarizer/window budgeting; making it call-time-lazy seemed too invasive for v1). Reasonable?

[CSOAI-ORG]

Hi @sklinglernv — this design matches something we measured independently and can share.

We built a router over specialist "lanes" for a governance/safety gate, each matched to a threat class and measured separately rather than relying on one frontier model for every method. Two lanes are quotable at n>=30 with disjoint confidence intervals against a frontier comparator:

• injection detection: a 200KB TF-IDF classifier vs gpt-4o-mini as judge, on a held-out split of the deepset prompt-injection corpus (0/546 training-row overlap). Our lane: recall 1.00 [0.938, 1.000], n=80. Comparator: recall ceiling 0.877 (upper CI bound). Exact two-sided McNemar test on the 12 discordant pairs: p = 0.00049.
• PII/identifier extraction: a 156-byte regex lens, recall 1.00 [0.901, 1.000] at n=35, 0/400 false-flag rate on benign text (upper CI bound 0.95%).

The reading we'd defend: routing beats a frontier judge on this specific threat class, not general capability — the frontier model becomes the fallback lane for anything the specialist lenses don't cover, rather than the default for everything. That's the same shape as Role("cheap") / Role("strong") here, just applied to a defense-gate use case.

Happy to open a small runnable example (deterministic core + narration, hermetic tests, matching the pattern from #75/#99) if that's useful evidence for the design discussion, or just leave this as a data point. All numbers above are from a signed decision ledger; can share the methodology/corpus split on request.