Skip to content

Commit 35989c8

Browse files
ClickerMonkeyclaude
andcommitted
feat(query): ship worked examples + sharper instructions on functions & nodes
LLM guidance is now first-class + reusable: FunctionDef.examples + a static EXAMPLES on Expr/Query classes (raw JSON strings), surfaced automatically by describeEngine with a maxExamples cap (DescribeEngineOptions.maxExamples, default 2; 0 omits). Populated the high-confusion set with sharper instructions + 1-2 worked examples each: the window family (rank/denseRank/lag/lead/firstValue/ lastValue/nthValue/cumeDist — each now spells out the partitionBy-vs-orderBy distinction the model kept botching), dateTrunc/age/replace/count; and the WindowExpr/Exists/In/Subquery/FunctionCall exprs + Select/SetOperation/CTE queries (the query classes gained INSTRUCTIONS + queryClassList()). Every shipped example is guarded by a test that parses it structurally via a bare registry (parseCheckedQuery/Expr) — a malformed shipped example fails CI. Migrated WORKED_EXAMPLE_QUERIES/exampleQueriesText onto the nodes (one source of truth); tool/cli/eval createAsker now get everything from describeEngine(engine, {maxExamples}). Library users get good prompts for free. 100% gate (1470 tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012aWFXmP2zN1wnX43EwRjBS
1 parent 9be3912 commit 35989c8

24 files changed

Lines changed: 627 additions & 307 deletions

packages/query/aeye-query.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ A `Type` is a named collection of `Field`s plus index + cardinality estimates (`
3737

3838
## Expression kinds
3939

40-
Every kind is one branch of the `ExprDef` union. Availability in the LLM schema is depth-graduated and capability-gated; the always-usable core is never gated. Each Expr class also exposes a concise `static INSTRUCTIONS` one-liner (enumerable via `registry.exprClassList()`) — the canonical terse doc mirrored by the table below.
40+
Every kind is one branch of the `ExprDef` union. Availability in the LLM schema is depth-graduated and capability-gated; the always-usable core is never gated. Each Expr class also exposes a concise `static INSTRUCTIONS` one-liner (enumerable via `registry.exprClassList()`) — the canonical terse doc mirrored by the table below — plus, on the high-confusion kinds, a `static EXAMPLES` set of RAW JSON strings that `describeEngine` renders (capped by `maxExamples`).
4141

4242
| kind | one-line meaning |
4343
| ---- | ---------------- |
@@ -134,7 +134,7 @@ The flat `TypeDef` the LLM sees can be arbitrarily richer behind the scenes. A `
134134

135135
## Function library
136136

137-
`createRegistry()` ships a default library (60+ functions across all four shapes), registered as `FunctionDef` (name + shape + **named** params + output) paired with a shape-tagged runtime. Calls use named args (`args: { paramName: <expr> }`). Register your own with `registerFunction` + `registerFunctionRun`. Introspect with `registry.functionList()`; get a promptable by-shape listing with `describeFunctions(engine)`. Every builtin `FunctionDef` carries a terse `instructions` one-liner (what it does / arg meaning / gotcha), surfaced on `QueryFunction.instructions`.
137+
`createRegistry()` ships a default library (60+ functions across all four shapes), registered as `FunctionDef` (name + shape + **named** params + output) paired with a shape-tagged runtime. Calls use named args (`args: { paramName: <expr> }`). Register your own with `registerFunction` + `registerFunctionRun`. Introspect with `registry.functionList()`; get a promptable by-shape listing with `describeFunctions(engine)`. Every builtin `FunctionDef` carries a terse `instructions` one-liner (what it does / arg meaning / gotcha), surfaced on `QueryFunction.instructions`; the high-confusion ones also carry `examples` (RAW JSON strings, round-tripped by `QueryFunction.from` / `toJSON`) that `describeEngine` renders under the signature.
138138

139139
Shapes: `scalar` `(args, ctx)→value`, `tabular` `(args, ctx)→rows`, `aggregate` `(rows, ctx)→value`, `window` `(partition, index, ctx)→value/row`. All builtin names are **camelCase**; where the emitted SQL name differs it is noted. The base (ANSI) dialect degrades where noted and never throws.
140140

@@ -273,7 +273,9 @@ The model-facing JSON-Schema (what `z.toJSONSchema` — and core's `compile()`
273273

274274
The `describe*` helpers render a compact, promptable capability summary (plain text, deliberately terse to protect the context budget):
275275

276-
- **`describeEngine(engine, { types?, functions? })`** composes ONE block a model can read to know everything it may use: every (supplied) Type, then `describeExprs`, then `describeFunctions`, then `describeDialects`. `functions` narrows both the expr gating and the function listing to the schema's selection; `types` narrows the Type list + gating.
277-
- **`describeExprs(engine, types?, functions?)`** lists the CAPABILITY-GATED expression kinds — one `kind — INSTRUCTIONS` line per kind actually usable for the current Types/functions, filtered by the SAME gate the schema uses (`exprKindApplicable`). The always-usable core is never gated; `semantic` / `text-search` / `text-score` / `array-op` / `relation-path` / `tabular-function-call` appear only when an eligible Type/function exists (`excluded` / `output` are position-only and never listed).
278-
- **`describeFunctions`** renders each function as `name(a, b?): output — instructions` (named params, a trailing `?` marks optional), grouped by shape.
276+
- **`describeEngine(engine, { types?, functions?, maxExamples? })`** composes ONE block a model can read to know everything it may use: every (supplied) Type, then `describeExprs`, then `describeFunctions`, then `describeQueryExamples`, then `describeDialects`. `functions` narrows both the expr gating and the function listing to the schema's selection; `types` narrows the Type list + gating; `maxExamples` caps how many WORKED examples render PER function / node (default `DEFAULT_MAX_EXAMPLES` = 2; `0` omits examples). It no longer needs a separate example-text call — worked examples now live ON the nodes/functions (see below) and are folded in here.
277+
- **`describeExprs(engine, types?, functions?, maxExamples?)`** lists the CAPABILITY-GATED expression kinds — one `kind — INSTRUCTIONS` line per kind actually usable for the current Types/functions, filtered by the SAME gate the schema uses (`exprKindApplicable`), each followed by up to `maxExamples` `e.g. <json>` lines from the kind's `static EXAMPLES`. The always-usable core is never gated; `semantic` / `text-search` / `text-score` / `array-op` / `relation-path` / `tabular-function-call` appear only when an eligible Type/function exists (`excluded` / `output` are position-only and never listed).
278+
- **`describeFunctions(engine, functions?, maxExamples?)`** renders each function as `name(a, b?): output — instructions` (named params, a trailing `?` marks optional), grouped by shape, each followed by up to `maxExamples` `e.g. <json>` lines from the function's `examples`.
279+
- **`describeQueryExamples(engine, maxExamples?)`** renders a query-examples section: each registered query KIND that ships `static EXAMPLES` (SELECT, UNION/INTERSECT/EXCEPT, WITH/CTE) as `kind — INSTRUCTIONS` plus up to `maxExamples` worked example queries. Query-level constructs have no expr-catalog entry, so they get their own section.
280+
- **Worked examples are a shipped, first-class surface.** Every `FunctionDef` may carry `examples?: readonly string[]` (RAW JSON strings — expr fragments or full queries that CALL it), and every Expr/Query node class may carry a `static readonly EXAMPLES?: readonly string[]` (plus, for query kinds, an optional `static INSTRUCTIONS`). Examples are TYPE-AGNOSTIC (illustrative generic source/field names like `event.score` / `user.name`) — they teach SHAPE while the catalog supplies the caller's real Type names. The shipped high-confusion set: the window family (`rank` / `denseRank` / `lag` / `lead` / `firstValue` / `lastValue` / `nthValue` / `cumeDist`), `dateTrunc` / `age` / `replace`, `count`; and the `window` / `exists` / `in` / `subquery` / `function-call` exprs and `select` / `union` / `cte` queries. This is the ONE source of truth — `describeEngine` only renders them. A structural test parses EVERY shipped example through a bare `createRegistry()`'s `parseCheckedQuery` / `parseCheckedExpr` and asserts zero problems.
279281
- **Generated Type / Field docs.** `describeType` / `describeField` always emit a short `label` + long `description`: the developer's `TypeDef.label` / `FieldDef.description` when set, otherwise a sensible default GENERATED on demand from the meta-model — a Field from its FieldType (kind, bounds, `sensitive` / `semantic` / `search` flags, array item/bounds, a relation's `to` + `count` → belongs-to / has-many, nullability), a Type from its name + field/relation/index summary. Read the (possibly-generated) pair directly with `fieldMeta(field)` / `typeMeta(type)` (`{ label, description }`); nothing mutates the stored def — the strings are computed fresh per call.

packages/query/examples/cli.ts

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,6 @@ import {
5656
describeTypes,
5757
describeFunctions,
5858
describeEngine,
59-
exampleQueriesText,
6059
depthInstructions,
6160
selectTypes,
6261
DEFAULT_MAX_QUERY_SCHEMA_TYPES,
@@ -345,7 +344,7 @@ async function createAsker(
345344
};
346345
const promptInstructions = (i: PromptInput): string =>
347346
i.engine
348-
? `${describeEngine(i.engine, { types: i.types, functions: CLI_FUNCTIONS })}\n\n${exampleQueriesText()}`
347+
? describeEngine(i.engine, { types: i.types, functions: CLI_FUNCTIONS, maxExamples: 2 })
349348
: '';
350349
const prompt = ai.prompt({
351350
name: 'query_build',

packages/query/integration/run.ts

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@ import {
3838
QueryToolError,
3939
querySchema,
4040
describeEngine,
41-
exampleQueriesText,
4241
type QueryEngine,
4342
type Query,
4443
type QueryDef,
@@ -295,7 +294,10 @@ function createAsker(apiKey: string, modelId: string, engine: QueryEngine): Quer
295294
// CONCEPTUAL value). No Tool is built: we parse directly with `parseQueryTool`.
296295
// Same boundary cast the tool applies to its own wire schema (see tool.ts).
297296
const wireSchema = querySchema(engine, options) as QuerySchema;
298-
const instructions = `${describeEngine(engine, { types, functions: 'all' })}\n\n${exampleQueriesText()}`;
297+
// `describeEngine` now folds worked examples (per expr kind, function, and query
298+
// kind) in from the nodes' own `EXAMPLES` — one source of truth. `maxExamples`
299+
// caps how many render per node/function.
300+
const instructions = describeEngine(engine, { types, functions: 'all', maxExamples: 2 });
299301

300302
// `parse` runs the query parser: returns the built Query, or the QueryToolError
301303
// whose `.message` (the compiler-style report) the prompt re-prompts with.

packages/query/src/__tests__/cov-llm-describe-generate.test.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,7 @@ describe('describeFunctions (enhanced) + describeEngine composition', () => {
211211

212212
it('renders named params (a, b?), output, and instructions', () => {
213213
const fns = describeFunctions(engine, 'all');
214-
expect(fns).toContain('count(value?): number — Count rows'); // optional param + instructions
214+
expect(fns).toContain('count(value?): number — Count ROWS'); // optional param + instructions
215215
expect(fns).toContain('sum(value): inferred — Sum of the non-null values.'); // inferred + instructions
216216
expect(fns).toContain('genRows(n): gadget'); // {type} output, no instructions
217217
});

packages/query/src/__tests__/cov-llm-describe.test.ts

Lines changed: 134 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -5,18 +5,35 @@
55
import { describe, it, expect } from 'vitest';
66
import { createRegistry, Registry } from '../registry';
77
import { QueryEngine } from '../engine';
8+
import { Problems } from '../problem';
9+
import { isRecord } from '../shape';
10+
import { ExprQuery } from '../queries/index';
811
import {
912
describeType,
1013
describeTypes,
1114
describeFunctions,
15+
describeExprs,
16+
describeQueryExamples,
1217
describeDialects,
1318
describeEngine,
14-
exampleQueriesText,
15-
WORKED_EXAMPLE_QUERIES,
19+
DEFAULT_MAX_EXAMPLES,
1620
} from '../llm/describe';
17-
import { createExampleFixture } from '../../examples/schema';
1821
import type { TypeDef } from '../schema';
1922

23+
/** Query `kind` discriminants — an example whose top-level `kind` is one of these
24+
* is a FULL query (validated via `parseCheckedQuery`); otherwise an expr fragment. */
25+
const QUERY_KINDS = new Set([
26+
'select',
27+
'insert',
28+
'update',
29+
'delete',
30+
'union',
31+
'intersect',
32+
'except',
33+
'cte',
34+
'expr',
35+
]);
36+
2037
const widgetDef: TypeDef = {
2138
name: 'widget',
2239
label: 'Widget',
@@ -89,25 +106,125 @@ describe('describeFunctions / describeDialects / describeEngine / describeTypes'
89106
const engine = widgetEngine();
90107
expect(describeTypes(engine)).toContain('## widget');
91108
expect(describeTypes(engine.registry)).toContain('## widget');
92-
expect(describeEngine(engine)).toContain('dialects:');
93-
expect(exampleQueriesText()).toContain('field-ref');
109+
const de = describeEngine(engine);
110+
expect(de).toContain('dialects:');
111+
expect(de).toContain('query examples:');
94112
});
95113
});
96114

97-
describe('WORKED_EXAMPLE_QUERIES', () => {
98-
it('every worked example parses + validates cleanly against the example fixture', () => {
99-
const { engine } = createExampleFixture();
100-
for (const [name, def] of Object.entries(WORKED_EXAMPLE_QUERIES)) {
101-
const problems = engine.validateQuery(engine.parseQuery(def));
102-
expect(problems.hasErrors, `${name}: ${problems.list.map((p) => p.code).join(', ')}`).toBe(false);
115+
/**
116+
* The SHIPPED examples (function `examples` + node `EXAMPLES`) are type-agnostic
117+
* and teach SHAPE, so they are validated STRUCTURALLY: each must `JSON.parse` and
118+
* pass a BARE registry's `parseCheckedQuery` (full-query examples) / `parseCheckedExpr`
119+
* (expr fragments) with ZERO structural problems. A malformed shipped example FAILS.
120+
*/
121+
describe('shipped examples are structurally valid', () => {
122+
const registry = createRegistry();
123+
124+
/** Structurally validate one raw-JSON example; discriminates query vs expr by `kind`. */
125+
function validate(raw: string): Problems {
126+
const parsed = JSON.parse(raw);
127+
const kind = isRecord(parsed) && typeof parsed['kind'] === 'string' ? parsed['kind'] : '';
128+
const p = new Problems();
129+
if (QUERY_KINDS.has(kind)) registry.parseCheckedQuery(parsed, p);
130+
else registry.parseCheckedExpr(parsed, p);
131+
return p;
132+
}
133+
134+
it('every FUNCTION example parses + validates with no structural problems', () => {
135+
let seen = 0;
136+
for (const fn of registry.functionList()) {
137+
for (const ex of fn.examples ?? []) {
138+
seen++;
139+
const p = validate(ex);
140+
expect(p.hasErrors, `${fn.name}: ${p.list.map((x) => x.code).join(', ')}${ex}`).toBe(false);
141+
}
142+
}
143+
expect(seen).toBeGreaterThan(0); // guard: the window family etc. ship examples
144+
});
145+
146+
it('every EXPR-node example parses + validates with no structural problems', () => {
147+
let seen = 0;
148+
for (const cls of registry.exprClassList()) {
149+
for (const ex of cls.EXAMPLES ?? []) {
150+
seen++;
151+
const p = validate(ex);
152+
expect(p.hasErrors, `${cls.KIND}: ${p.list.map((x) => x.code).join(', ')}${ex}`).toBe(false);
153+
}
154+
}
155+
expect(seen).toBeGreaterThan(0); // guard: window / exists / in / subquery / function-call ship examples
156+
});
157+
158+
it('every QUERY-node example parses + validates with no structural problems', () => {
159+
let seen = 0;
160+
for (const cls of registry.queryClassList()) {
161+
for (const ex of cls.EXAMPLES ?? []) {
162+
seen++;
163+
const p = validate(ex);
164+
expect(p.hasErrors, `${cls.KIND}: ${p.list.map((x) => x.code).join(', ')}${ex}`).toBe(false);
165+
}
103166
}
167+
expect(seen).toBeGreaterThan(0); // guard: select / union / cte ship examples
168+
});
169+
});
170+
171+
describe('describeEngine example rendering + maxExamples', () => {
172+
const engine = widgetEngine();
173+
174+
it('renders worked examples under exprs, functions, and the query-examples section', () => {
175+
const de = describeEngine(engine);
176+
// Expr-kind example (WindowExpr ships a worked rank SELECT). Rendered examples
177+
// begin `e.g. {` — distinct from the literal "e.g." some INSTRUCTIONS contain.
178+
expect(de).toContain(' - window —');
179+
expect(de).toContain('e.g. {');
180+
expect(de).toContain('"function":"rank"');
181+
// Query-examples section (SetOperationQuery / CTEStatementQuery / SelectQuery).
182+
expect(de).toContain('query examples:');
183+
expect(de).toContain('"kind":"union"');
184+
expect(de).toContain('"kind":"cte"');
185+
});
186+
187+
it('maxExamples caps examples per node / function; 0 omits them entirely', () => {
188+
const none = describeEngine(engine, { maxExamples: 0 });
189+
expect(none).not.toContain('e.g. {');
190+
// The query-examples section header still renders (kinds + instructions), sans examples.
191+
expect(none).toContain('query examples:');
192+
193+
// InExpr ships TWO examples; a cap of 1 shows only the first (the value LIST form).
194+
const capped = describeExprs(engine, undefined, 'all', 1);
195+
const inLine = capped.split('\n').filter((l) => l.includes('"kind":"in"'));
196+
expect(inLine.length).toBe(1);
197+
const full = describeExprs(engine, undefined, 'all', 2);
198+
expect(full.split('\n').filter((l) => l.includes('"kind":"in"')).length).toBe(2);
199+
});
200+
201+
it('describeQueryExamples renders only kinds that ship EXAMPLES', () => {
202+
const qe = describeQueryExamples(engine);
203+
expect(qe).toContain('query examples:');
204+
expect(qe).toContain(' select —');
205+
expect(qe).toContain(' union —');
206+
expect(qe).toContain(' cte —');
207+
// insert/update/delete/expr ship no examples ⇒ absent.
208+
expect(qe).not.toContain(' insert');
209+
// An empty registry yields the "(none)" sentinel.
210+
expect(describeQueryExamples(new Registry())).toBe('query examples: (none)');
211+
});
212+
213+
it('renders a query kind that ships EXAMPLES but no INSTRUCTIONS (no em-dash)', () => {
214+
const reg = createRegistry();
215+
// Override the `expr` kind with an entry that ships EXAMPLES but NO INSTRUCTIONS.
216+
reg.defineQuery({
217+
KIND: 'expr',
218+
from: ExprQuery.from,
219+
EXAMPLES: ['{"kind":"expr","expr":{"kind":"literal","value":1}}'],
220+
});
221+
const line = describeQueryExamples(reg)
222+
.split('\n')
223+
.find((l) => l.startsWith(' expr'));
224+
expect(line).toBe(' expr'); // no ` — <INSTRUCTIONS>` suffix
104225
});
105226

106-
it('exampleQueriesText embeds the worked examples (window / union / cte / exists)', () => {
107-
const text = exampleQueriesText();
108-
expect(text).toContain('"kind": "union"');
109-
expect(text).toContain('"kind": "cte"');
110-
expect(text).toContain('"kind": "exists"');
111-
expect(text).toContain('"function": "rank"');
227+
it('DEFAULT_MAX_EXAMPLES is a small positive cap', () => {
228+
expect(DEFAULT_MAX_EXAMPLES).toBeGreaterThan(0);
112229
});
113230
});

packages/query/src/__tests__/self-describing.test.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ describe('self-describing catalog', () => {
3737
expect(fn.instructions).toBe(def.instructions);
3838
const json = fn.toJSON();
3939
expect(json.instructions).toBe(def.instructions);
40+
expect(fn.examples).toEqual(def.examples);
41+
expect(json.examples).toEqual(def.examples);
4042
expect(json.name).toBe(def.name);
4143
expect(json.shape).toBe(def.shape);
4244
expect(json.params).toEqual(def.params);

packages/query/src/expr.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,13 @@ export interface ExprClass {
102102
* (`registry.exprClassList()`) to build a self-describing expr catalog.
103103
*/
104104
readonly INSTRUCTIONS: string;
105+
/**
106+
* OPTIONAL worked examples — each a RAW JSON string (an expr fragment OR a full
107+
* query that USES this kind) teaching its SHAPE with illustrative generic
108+
* source/field names. Surfaced (capped by `maxExamples`) under this kind's
109+
* catalog entry by `describeEngine`. The ONE source of truth for the examples.
110+
*/
111+
readonly EXAMPLES?: readonly string[];
105112
/** Build an instance from its JSON branch, recursing into child defs via `registry.parseExpr`. */
106113
from(json: ExprDef, registry: Registry): Expr;
107114
/** Zod schema for this expr kind's JSON `ExprDef` branch. */

0 commit comments

Comments
 (0)