pywhyllm - v2

[grace-sng7]

---

Summary of Changes

---

LLM backend: moved from guidance to instructor

Replaced the guidance library with instructor. The old design in v0.1 used blocking retry loops and parsed results from the LLM. The new design gets typed Pydantic objects directly from any instructor-wrapped async client. instructor also enables compatibility with more LLMs.

---

Model suggester class structure

ModelSuggester, IdentificationSuggester, and ValidationSuggester were three separately-instantiated classes. They are now three private mixins (DiscoveryMixin, IdentificationMixin, ValidationMixin) that compose into a single ModelSuggester.

IdentificationSuggester, ValidationSuggester, and SimpleIdentificationSuggester are deleted as they are subsumed by the ModelSuggester.

---

API Call Reduction

The original suggest_relationships called suggest_pairwise_relationship once per variable pair, per expert — sequentially. The new suggest_graph sends all variables in a single prompt, and with multiple experts fires all calls in parallel via the asyncio library.

Below illustrates the change in the number of API calls when using the suggest_graph method.

Scenario	Before	After
20 variables, no experts	~190 sequential calls	1 call
20 variables, 3 experts	~570 sequential calls	3 parallel calls

We further reduce the number of API calls needed in the entire workflow with the next change.

---

Return types: raw lists → CausalGraph object

suggest_relationships returned a raw list of (cause, effect) tuples with no metadata, and structural queries (parents, children, mediators, IVs) all required additional LLM calls.

The new CausalGraph object stores per-edge vote counts, confidence scores, and per-expert reasonings, and answers all structural queries locally with zero LLM calls:

• parents_of, children_of, ancestors_of, descendants_of
• mediators_of, instrumental_variables_for
• top_edges, edge_data, reasoning_for
• plot() — built-in Graphviz DAG visualization

---

Other changes

• min_confidence threshold (default 0.5) — each expert provides a confidence level (from 0-1) for their reasoning. "Low-confidence" edges are dropped.
• _prompts.py — all prompt strings extracted into 8 builder functions so that they aren't hardcoded into methods
• Provider-agnostic — accepts any instructor-wrapped async client; switching providers requires only one line

---

Tests

Added test_causal_graph.py (33 tests), test_prompts.py (20 tests), expanded test_model_suggester.py (25 async tests). Removed test files for deleted classes.

---

Significant changes

• ModelSuggester(llm=guidance_model) → ModelSuggester(client=instructor_client, model="...")
• suggest_relationships(...) which previously returned list[tuple] now returns a CausalGraph
• suggest_parents/children/mediators/ivs are now zero-LLM-call methods on CausalGraph
• IdentificationSuggester and ValidationSuggester are removed; all functionality is on ModelSuggester

---