Skip to content

Commit 322aaed

Browse files
philcunliffeclaude
andauthored
Context-graph plugin: T0 activity-graph projection over ai_gateway_messages (#97)
* Context-graph plugin: T0 activity-graph projection over ai_gateway_messages The first slice of the context-graph direction (designed in the cgproto LLP corpus, LLP 0006 "Projection pipeline"): a deterministic T0 projection that turns recorded gateway traffic into a queryable node/edge activity graph. No models involved — T0 is pure relational projection over data the gateway already structures. The @hypaware/context-graph plugin registers: - Datasets `node` and `edge` — derived Iceberg tables fronted by the kernel cache, queryable via `hyp query sql`. - `hyp graph project [--dry-run]` — runs 9 hand-authored contract rules over ai_gateway_messages, materializing 5 node types (Session, App, Model, Tool, File) and 4 edge types (via, used_model, used, touched). Ids are content-addressed (SHA-256 of type + natural key) and every row carries inline provenance (source_dataset, source_keys, projector, projector_version), so re-projection is idempotent: pre-write dedup filters ids already committed, and a re-run writes zero rows. - `hyp graph compact [--dry-run]` — merges duplicate node/edge rows that slip past pre-write dedup (concurrent projections, partial failures), possibly across `source=` partitions: each duplicate group folds into one row (earliest first_seen, unioned props — the same mergeRow projection uses) kept in the earliest-seen partition, and affected partitions are rewritten via the cache's generation swap (new table dir, cursor repoint, .retired marker for the kernel sweep). Kernel-side enablers (shared, not graph-specific in shape): - `AppendOptions.sortOrder`: a column-name sort declaration applied at table creation; icebird >= 0.8.9 then sorts every appended data file by the table's default sort order. Graph rewrites declare (node_type, node_id) / (edge_type, src_id, dst_id) so type scans and id lookups prune after the first compaction. - Cache compaction (compactSourceTable) now carries an existing default sort order over to its replacement table — previously the generation swap silently dropped it. - `@hypaware/context-graph` added to the bundled-plugin allowlist. The contract is hand-authored per rule for now; the declarative contract -> SQL compiler is a later slice (cgproto LLP 0006 "projection contracts compile to SQL"). Covered by test/plugins/context-graph-maintenance.test.js (cross- partition dedup merge, sort-order declaration on rewrite, idempotence) and the context_graph_projects_rows hermetic smoke (projection counts, node_type breakdown, idempotent re-run, clean compaction round-trip through real plugin registration). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address dual-review findings for PR #97 - maintenance.js: positively read cursors (tryReadCursorSync) and require source-table layout before any rewrite; make the generation swap conditional on the cursor matching the scan-time read so rows appended during the rewrite window are never lost — on mismatch the staged table is removed and the partition is reported skipped, never retired. Home partitions rewrite before copy-droppers and copies are only dropped once the merged row landed. Skips surface in the report, on stderr, and as span attributes; unreadable cursors exit nonzero. - project.js: dedup query failures now abort the projection unless the dataset is genuinely missing; mergeRow resolves props conflicts deterministically (per-key earliest-seen wins, value tie-break) so merge order can never change the result. - datasets.js: unionSources no longer forwards limit/offset to sub-sources (offsets were applied twice on multi-partition datasets). - ids.js: literal NUL bytes replaced with \0 escapes (byte-identical hash input, file is plain text again); delimiter choice documented. - format-iceberg maintenance.js: compactExportTable wrapped in a sink.export.compact span recording reason, file counts, commit verification outcome, staged-file reclamation, and error_kind. - hypaware.plugin.json: declare 'graph compact' in contributes.commands. - smoke: context_graph_projects_rows now asserts the graph.project / graph.compact spans alongside the SQL counts. - tests: pinned nodeId/edgeId digests, contract toRow rules, mergeRow determinism, union limit/offset, corrupt-cursor and concurrent-write compaction safety, CLI skip reporting. - LLP 0023 documents the context-graph T0 projection decisions; doc map updated and design comments converted to @ref annotations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent dade52a commit 322aaed

22 files changed

Lines changed: 2434 additions & 3 deletions
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"schema_version": 1,
3+
"name": "@hypaware/context-graph",
4+
"version": "0.1.0",
5+
"description": "T0 deterministic projection: materializes a node/edge activity graph from ai_gateway_messages. Reads structured capture, emits graph rows with inline provenance — no models, exact-key matching only.",
6+
"hypaware_api": "^1.0.0",
7+
"runtime": "node",
8+
"node_engine": ">=20",
9+
"entrypoint": "./src/index.js",
10+
"permissions": ["read_state", "write_state"],
11+
"contributes": {
12+
"datasets": [{ "name": "node" }, { "name": "edge" }],
13+
"commands": [{ "name": "graph project" }, { "name": "graph compact" }]
14+
}
15+
}
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
// @ts-check
2+
3+
import { compactGraphTables } from './maintenance.js'
4+
import { projectGraph } from './project.js'
5+
6+
/**
7+
* @import { CommandRunContext } from '../../../../collectivus-plugin-kernel-types.d.ts'
8+
* @import { ExtendedQueryStorageService } from '../../../../src/core/cache/types.d.ts'
9+
*/
10+
11+
/**
12+
* `hyp graph project` — run the T0 projection over `ai_gateway_messages`.
13+
*
14+
* @param {string[]} argv
15+
* @param {CommandRunContext} ctx
16+
* @returns {Promise<number>}
17+
*/
18+
export async function runGraphProject(argv, ctx) {
19+
const dryRun = argv.includes('--dry-run')
20+
try {
21+
const r = await projectGraph({
22+
query: ctx.query,
23+
storage: /** @type {ExtendedQueryStorageService} */ (ctx.storage),
24+
config: ctx.config,
25+
dryRun,
26+
})
27+
if (dryRun) {
28+
ctx.stdout.write(`graph project (dry-run): ${r.nodes} node(s), ${r.edges} edge(s) would be projected\n`)
29+
} else {
30+
ctx.stdout.write(
31+
`graph project: ${r.nodes} node(s), ${r.edges} edge(s) — wrote ${r.nodesWritten} new node(s), ${r.edgesWritten} new edge(s)\n`
32+
)
33+
}
34+
return 0
35+
} catch (err) {
36+
ctx.stderr.write(`hyp graph project: ${err instanceof Error ? err.message : String(err)}\n`)
37+
return 1
38+
}
39+
}
40+
41+
/**
42+
* `hyp graph compact` — merge duplicate node/edge rows and rewrite
43+
* affected partitions into sorted replacement tables.
44+
*
45+
* @param {string[]} argv
46+
* @param {CommandRunContext} ctx
47+
* @returns {Promise<number>}
48+
*/
49+
export async function runGraphCompact(argv, ctx) {
50+
const dryRun = argv.includes('--dry-run')
51+
try {
52+
const r = await compactGraphTables({
53+
storage: /** @type {ExtendedQueryStorageService} */ (ctx.storage),
54+
dryRun,
55+
})
56+
for (const d of r.datasets) {
57+
if (dryRun) {
58+
ctx.stdout.write(
59+
`graph compact (dry-run): ${d.dataset}${d.duplicateIds} duplicate id(s) across ${d.partitionsRewritten} partition(s) would be merged\n`
60+
)
61+
} else {
62+
ctx.stdout.write(
63+
`graph compact: ${d.dataset} — merged ${d.rowsMerged} duplicate row(s) (${d.duplicateIds} id(s)), rewrote ${d.partitionsRewritten} partition(s)\n`
64+
)
65+
}
66+
for (const skip of d.partitionsSkipped) {
67+
ctx.stderr.write(`hyp graph compact: skipped ${skip.path} (${skip.reason})\n`)
68+
}
69+
}
70+
// A concurrent-write skip is a benign retry-later; an unreadable
71+
// cursor needs operator attention — exit nonzero so it can't pass
72+
// silently in scripts.
73+
const unreadable = r.datasets.some((d) => d.partitionsSkipped.some((s) => s.reason === 'unreadable-cursor'))
74+
return unreadable ? 1 : 0
75+
} catch (err) {
76+
ctx.stderr.write(`hyp graph compact: ${err instanceof Error ? err.message : String(err)}\n`)
77+
return 1
78+
}
79+
}
Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
// @ts-check
2+
3+
import path from 'node:path'
4+
5+
import { nodeId, edgeId } from './ids.js'
6+
7+
/**
8+
* The hand-authored T0 contract for `ai_gateway_messages`. Each rule is a
9+
* read-only SELECT (the contract's read half is genuinely SQL) plus a
10+
* `toRow` that maps a result row to a graph node/edge with deterministic
11+
* id and inline provenance. `toRow` returns `null` to skip a row.
12+
*
13+
* A generic declarative-contract -> SQL compiler is a later slice; for now
14+
* the rules are explicit and each SELECT documents the structural fact it
15+
* extracts.
16+
*
17+
* @import { GraphRow, ContractRule } from './types.d.ts'
18+
*/
19+
20+
export const SOURCE_DATASET = 'ai_gateway_messages'
21+
export const PROJECTOR = 'ai-gateway.t0'
22+
export const PROJECTOR_VERSION = 1
23+
24+
/** Tools whose args name a concrete file. */
25+
const FILE_TOOLS = new Set(['Read', 'Edit', 'Write', 'MultiEdit', 'NotebookEdit'])
26+
27+
/**
28+
* @param {{ type: string, key: string, label?: string | null, props?: Record<string, unknown>, firstSeen: unknown, sourceKeys: Record<string, unknown> }} spec
29+
* @returns {GraphRow}
30+
*/
31+
function buildNode(spec) {
32+
return {
33+
node_id: nodeId(spec.type, spec.key),
34+
node_type: spec.type,
35+
natural_key: spec.key,
36+
label: spec.label ?? null,
37+
props: spec.props && Object.keys(spec.props).length > 0 ? spec.props : null,
38+
first_seen: firstSeen(spec.firstSeen),
39+
source_dataset: SOURCE_DATASET,
40+
source_keys: spec.sourceKeys,
41+
projector: PROJECTOR,
42+
projector_version: PROJECTOR_VERSION,
43+
}
44+
}
45+
46+
/**
47+
* @param {{ type: string, srcType: string, srcKey: string, dstType: string, dstKey: string, firstSeen: unknown, sourceKeys: Record<string, unknown> }} spec
48+
* @returns {GraphRow}
49+
*/
50+
function buildEdge(spec) {
51+
const src = nodeId(spec.srcType, spec.srcKey)
52+
const dst = nodeId(spec.dstType, spec.dstKey)
53+
return {
54+
edge_id: edgeId(src, spec.type, dst),
55+
edge_type: spec.type,
56+
src_id: src,
57+
dst_id: dst,
58+
src_type: spec.srcType,
59+
dst_type: spec.dstType,
60+
props: null,
61+
first_seen: firstSeen(spec.firstSeen),
62+
source_dataset: SOURCE_DATASET,
63+
source_keys: spec.sourceKeys,
64+
projector: PROJECTOR,
65+
projector_version: PROJECTOR_VERSION,
66+
}
67+
}
68+
69+
/**
70+
* The T0 rules. Node rules first, then edge rules.
71+
*
72+
* @type {ReadonlyArray<ContractRule>}
73+
* @ref LLP 0023#t0-contract [implements] — hand-authored rule list; a declarative-contract compiler is a deliberate later slice
74+
*/
75+
export const CONTRACT_RULES = Object.freeze([
76+
// --- nodes ---
77+
78+
// Session per conversation. SQL: SELECT conversation_id, ... (one row per part)
79+
{
80+
kind: 'node',
81+
type: 'Session',
82+
sql: `SELECT conversation_id, cwd, git_branch, client_name, user_id, message_created_at FROM ${SOURCE_DATASET}`,
83+
toRow(r) {
84+
const key = str(r.conversation_id)
85+
if (!key) return null
86+
return buildNode({
87+
type: 'Session',
88+
key,
89+
props: pruned({ cwd: str(r.cwd), git_branch: str(r.git_branch), client_name: str(r.client_name), user_id: str(r.user_id) }),
90+
firstSeen: r.message_created_at,
91+
sourceKeys: { conversation_id: key },
92+
})
93+
},
94+
},
95+
96+
// App per client_name.
97+
{
98+
kind: 'node',
99+
type: 'App',
100+
sql: `SELECT client_name, message_created_at FROM ${SOURCE_DATASET}`,
101+
toRow(r) {
102+
const key = str(r.client_name)
103+
if (!key) return null
104+
return buildNode({ type: 'App', key, label: key, firstSeen: r.message_created_at, sourceKeys: { client_name: key } })
105+
},
106+
},
107+
108+
// Model per model id.
109+
{
110+
kind: 'node',
111+
type: 'Model',
112+
sql: `SELECT model, message_created_at FROM ${SOURCE_DATASET}`,
113+
toRow(r) {
114+
const key = str(r.model)
115+
if (!key) return null
116+
return buildNode({ type: 'Model', key, label: key, firstSeen: r.message_created_at, sourceKeys: { model: key } })
117+
},
118+
},
119+
120+
// Tool per tool_name, from tool_call parts.
121+
{
122+
kind: 'node',
123+
type: 'Tool',
124+
sql: `SELECT tool_name, message_created_at FROM ${SOURCE_DATASET} WHERE part_type = 'tool_call'`,
125+
toRow(r) {
126+
const key = str(r.tool_name)
127+
if (!key) return null
128+
return buildNode({ type: 'Tool', key, label: key, firstSeen: r.message_created_at, sourceKeys: { tool_name: key } })
129+
},
130+
},
131+
132+
// File per resolved path, from file-touching tool calls.
133+
{
134+
kind: 'node',
135+
type: 'File',
136+
sql: `SELECT tool_name, tool_args, message_created_at FROM ${SOURCE_DATASET} WHERE part_type = 'tool_call'`,
137+
toRow(r) {
138+
const file = filePathFrom(r.tool_name, r.tool_args)
139+
if (!file) return null
140+
return buildNode({ type: 'File', key: file, label: path.basename(file), firstSeen: r.message_created_at, sourceKeys: { file_path: file } })
141+
},
142+
},
143+
144+
// --- edges ---
145+
146+
// Session -via-> App
147+
{
148+
kind: 'edge',
149+
type: 'via',
150+
sql: `SELECT conversation_id, client_name, message_created_at FROM ${SOURCE_DATASET}`,
151+
toRow(r) {
152+
const session = str(r.conversation_id)
153+
const app = str(r.client_name)
154+
if (!session || !app) return null
155+
return buildEdge({ type: 'via', srcType: 'Session', srcKey: session, dstType: 'App', dstKey: app, firstSeen: r.message_created_at, sourceKeys: { conversation_id: session, client_name: app } })
156+
},
157+
},
158+
159+
// Session -used_model-> Model
160+
{
161+
kind: 'edge',
162+
type: 'used_model',
163+
sql: `SELECT conversation_id, model, message_created_at FROM ${SOURCE_DATASET}`,
164+
toRow(r) {
165+
const session = str(r.conversation_id)
166+
const model = str(r.model)
167+
if (!session || !model) return null
168+
return buildEdge({ type: 'used_model', srcType: 'Session', srcKey: session, dstType: 'Model', dstKey: model, firstSeen: r.message_created_at, sourceKeys: { conversation_id: session, model } })
169+
},
170+
},
171+
172+
// Session -used-> Tool
173+
{
174+
kind: 'edge',
175+
type: 'used',
176+
sql: `SELECT conversation_id, tool_name, message_created_at FROM ${SOURCE_DATASET} WHERE part_type = 'tool_call'`,
177+
toRow(r) {
178+
const session = str(r.conversation_id)
179+
const tool = str(r.tool_name)
180+
if (!session || !tool) return null
181+
return buildEdge({ type: 'used', srcType: 'Session', srcKey: session, dstType: 'Tool', dstKey: tool, firstSeen: r.message_created_at, sourceKeys: { conversation_id: session, tool_name: tool } })
182+
},
183+
},
184+
185+
// Session -touched-> File
186+
{
187+
kind: 'edge',
188+
type: 'touched',
189+
sql: `SELECT conversation_id, tool_name, tool_args, message_created_at FROM ${SOURCE_DATASET} WHERE part_type = 'tool_call'`,
190+
toRow(r) {
191+
const session = str(r.conversation_id)
192+
const file = filePathFrom(r.tool_name, r.tool_args)
193+
if (!session || !file) return null
194+
return buildEdge({ type: 'touched', srcType: 'Session', srcKey: session, dstType: 'File', dstKey: file, firstSeen: r.message_created_at, sourceKeys: { conversation_id: session, file_path: file } })
195+
},
196+
},
197+
])
198+
199+
/**
200+
* Resolve a file path from a file-touching tool's args. `tool_args` is a
201+
* JSON column that may arrive parsed or as a string.
202+
*
203+
* @param {unknown} toolName
204+
* @param {unknown} toolArgs
205+
* @returns {string | null}
206+
*/
207+
function filePathFrom(toolName, toolArgs) {
208+
const name = str(toolName)
209+
if (!name || !FILE_TOOLS.has(name)) return null
210+
const args = parseMaybeJson(toolArgs)
211+
if (!args || typeof args !== 'object') return null
212+
const obj = /** @type {Record<string, unknown>} */ (args)
213+
return str(obj.file_path) ?? str(obj.notebook_path) ?? null
214+
}
215+
216+
/**
217+
* @param {unknown} value
218+
* @returns {unknown}
219+
*/
220+
function parseMaybeJson(value) {
221+
if (typeof value !== 'string') return value
222+
try {
223+
return JSON.parse(value)
224+
} catch {
225+
return null
226+
}
227+
}
228+
229+
/**
230+
* @param {unknown} value
231+
* @returns {string | null}
232+
*/
233+
function str(value) {
234+
if (typeof value === 'string') return value.length > 0 ? value : null
235+
if (typeof value === 'number' || typeof value === 'bigint') return String(value)
236+
return null
237+
}
238+
239+
/**
240+
* Normalize a timestamp-ish value to an ISO string when possible.
241+
*
242+
* @param {unknown} value
243+
* @returns {string | null}
244+
*/
245+
function firstSeen(value) {
246+
if (typeof value === 'string') return value.length > 0 ? value : null
247+
if (value instanceof Date) return value.toISOString()
248+
if (typeof value === 'number' && Number.isFinite(value)) return new Date(value).toISOString()
249+
return null
250+
}
251+
252+
/**
253+
* Drop null/undefined entries so identical inputs build identical props.
254+
*
255+
* @param {Record<string, unknown>} obj
256+
* @returns {Record<string, unknown>}
257+
*/
258+
function pruned(obj) {
259+
/** @type {Record<string, unknown>} */
260+
const out = {}
261+
for (const key of Object.keys(obj).sort()) {
262+
if (obj[key] != null) out[key] = obj[key]
263+
}
264+
return out
265+
}

0 commit comments

Comments
 (0)