SY-3833: Strongly Type Schematics on the Core

[emilbon99]

Issue Pull Request

Linear Issue

SY-3833

Description

This is the next split in the SY-3833 schematic strong-typing series. It moves the schematic from an opaque data: record blob into a fully typed shape on the server (Legend, Node, Edge with nested Handle, EdgeProps, structured props/legend/authority), and brings server-side migration support to parity with what the console already has.

Highlights:

• schemas/schematic.oracle rewritten as the strongly-typed shape; client TS, Go, and protobuf types regenerate from it. Console v6 state types now alias the oracle-generated nodeZ/edgeZ/legendZ/edgePropsZ/handleZ/segmentZ so the wire format and console state can never drift.
• One-time legacy unwrap chain at core/pkg/service/schematic/migrations/legacy/v0..v5/. Each version owns its own Lift step that decodes the blob if its own version matches, otherwise recurses into the predecessor. The top-level MigrateSchematic is a normal typed-to-typed step (AutoMigrateSchematic for gorp-entry fields + a single inline projection from v5.Data to the typed body).
• v55_lift_typed_schematic wired into the gorp Migrations slice (sequenced after msgpack_to_orc) so existing clusters lift their stored schematics on first read after the upgrade.
• Generator extensions in oracle/plugin/go/:
  • Codec generator emits make(map[string]msgpack.EncodedJSON, n) for map<string, record> fields (was emitting interface{}).
  • Protobuf translator handles non-primitive map values: record via structpb.NewStruct/AsMap, struct values via per-element XToPB/XFromPB, with a new MapValueConversion.{Forward,Backward}HasError template branch for error-returning conversions.
  • Added int16/uint16 cases to the primitive-conversion table so int16? fields round-trip through proto's int32.
  • Test fixture generator emits msgpack.EncodedJSON{...} literals for record so generated codec tests compile.
• SetData(ctx, key, Schematic) now takes a typed body, preserves Key and Name from the existing entry, and refuses snapshot edits. The TS client (SetDataBody) and console call sites (Schematic.tsx, services/link.ts, services/ontology.tsx, range/overview/Snapshots.tsx, workspace/services/ontology.tsx, export.ts) project console state into the typed shape directly.
• Slice fix: applyEdgeChanges now initializes props[edge.key] = {} on every add so newly-drawn edges always have a parseable EdgeProps entry. Selectors (selectNodeProps, selectElementProps, selectEdgeProps) safe-parse against the loose props record and return undefined on absence/mismatch instead of throwing.
• Schema tweaks vs. sy-3833 inspiration:
  • Node.measured dropped (pluto re-measures at render time; storing it server-side was dead weight).
  • Node.z_index is int16? (right-sized, optional to match v0 wire format).
  • EdgeProps.variant uses @validate default "pipe", EdgeProps.segments defaults to [], so edgePropsZ.parse({}) produces valid defaults — used both at edge-add time and during the v5→v6 lift.

Test coverage: 25 schematic specs + 2 v55 gorp e2e specs + 26 symbol specs all pass; per-version legacy migration tests exercise the v0→v5 chain through realistic blob fixtures.

Basic Readiness

• I have performed a self-review of my code.
• I have added relevant, automated tests to cover the changes.
• I have updated documentation to reflect the changes.

Greptile Summary

This PR strongly-types the server-side schematic from an opaque data: record blob into a fully structured Legend, Node, Edge, EdgeProps, and Handle schema, generated from schematic.oracle. It also adds a full v0→v6 server-side migration chain (MigrateSchematic), wires a v55_lift_typed_schematic gorp migration, and updates the TypeScript console/client to project directly against the new typed shape rather than spreading an untyped data bag. Previously flagged issues (duplicate migrateLegendColors, missing units in ZERO_LEGEND, ZIndex cast, nil Colors/Props maps, Snapshot overwrite, remoteCreated omissions) all appear resolved in this revision.

Confidence Score: 5/5

Safe to merge — all previously flagged P0/P1 issues are resolved in this revision.

All P0/P1 issues from prior review rounds (duplicate migrateLegendColors compile error, missing units in ZERO_LEGEND, ZIndex int16/int32 cast, nil Colors and Props map serialisation, Snapshot overwrite in SetData, remoteCreated omissions across call sites) are addressed. Migration chain is well-tested with per-version fixture data. No new P0/P1 issues found.

No files require special attention. The v6.ts stateMigration legend fallback (spreading undefined when state.legend is absent) is a known pre-existing edge case already noted in prior threads.

Important Files Changed

Filename	Overview
core/pkg/service/schematic/migrate.go	New file: MigrateSchematic lifts v55 blobs to typed Schematic; migrateLegend initialises Colors map, MigrateSchematic guards nil Props — previously flagged null-serialisation issues are resolved.
core/pkg/service/schematic/pb/translator.gen.go	Expanded translator: ZIndex cast int32(r.ZIndex)/int16(pb.ZIndex) is present and correct; Props round-trips through structpb opaque map; no nil-deref issues.
console/src/schematic/types/v6.ts	Aliases oracle-generated types; single migrateLegendColors declaration (compile-error duplicate resolved); ZERO_LEGEND has correct units; legend migration may produce invalid shape when v5 state.legend is absent but this was already flagged in prior threads.
client/ts/src/schematic/client.ts	SetData now typed to SetDataBody; ZERO_LEGEND correctly sets units: { x: 'px', y: 'px' } (previously flagged 5000%-position issue is resolved).
console/src/schematic/Schematic.tsx	navigateToLinkedSchematic and useLoadRemote both pass remoteCreated: true; useSyncComponent projects all typed fields correctly.
core/pkg/service/schematic/service.go	Wires msgpack_to_orc → v55_lift_typed_schematic migration chain via migrate.WithAddedDeps; looks correct.
core/pkg/service/schematic/writer.go	SetData preserves Key, Name, and Snapshot from stored entry — Snapshot overwrite issue from prior review is resolved; writer_test covers both fields-preserved and snapshot-preserved cases.
core/pkg/service/schematic/migrations/legacy/legacy.go	Clean dispatch table for v0–v5 + orphan-edge filter; decode handles nil blob as zero-value T correctly.
console/src/schematic/selectors.ts	selectNodeProps/selectEdgeProps/selectElementProps now use safeParse returning undefined on failure instead of unsafe cast — correct defensive pattern.
console/src/schematic/slice.ts	applyEdgeChanges initialises props[edge.key] = {} on add; fixThemeContrast now uses safeParse for color validation — both correctly address previously identified patterns.

Sequence Diagram

sequenceDiagram
    participant Console
    participant TSClient as TS Client
    participant API as Go API
    participant Writer
    participant GorpTable as Gorp Table
    participant Migration as v55 Migration

    Note over GorpTable,Migration: First read after upgrade
    GorpTable->>Migration: msgpack_to_orc (codec change)
    Migration->>Migration: v55_lift_typed_schematic
    Migration->>Migration: legacy.MigrateData (v0→v5 chain)
    Migration->>Migration: MigrateSchematic (v5→typed Schematic)
    Migration-->>GorpTable: Schematic{Nodes, Edges, Props, Legend}

    Note over Console,API: Create / Sync flow
    Console->>TSClient: schematics.create({nodes, edges, props, legend, ...})
    TSClient->>API: POST /schematic/create
    API->>Writer: Create(ctx, ws, &Schematic)
    Writer-->>GorpTable: Store typed Schematic

    Note over Console,API: SetData flow
    Console->>TSClient: schematics.setData(key, SetDataBody)
    TSClient->>API: POST /schematic/set-data
    API->>Writer: SetData(ctx, key, Schematic)
    Writer->>Writer: preserve Key, Name, Snapshot
    Writer-->>GorpTable: Updated Schematic

    Note over Console,API: Retrieve flow
    GorpTable-->>API: Schematic{Nodes, Edges, Props, Legend}
    API-->>TSClient: schematicZ-validated response
    TSClient-->>Console: Schematic (typed)
    Console->>Console: internalCreate({...ZERO_STATE, ...v, remoteCreated:true})

Loading

Comments Outside Diff (2)

1. console/src/schematic/types/v6.ts, line 157-167 (link)

   Duplicate migrateLegendColors declaration — compile error

   migrateLegendColors is declared twice as a const in the same module scope (lines 145 and 157). TypeScript will reject this with Cannot redeclare block-scoped variable 'migrateLegendColors', preventing the console from building. The second declaration (returning LegendState["colors"]) is the old version that should have been removed when the type was updated to use Legend["colors"].

2. client/ts/src/schematic/client.ts, line 156-160 (link)

   ZERO_LEGEND position renders at 5000% when units absent

   ZERO_LEGEND.position is { x: 50, y: 50 } with no units. The sticky-XY renderer at x/ts/src/spatial/sticky/sticky.ts interprets missing units as decimal fractions: ${pos.x * 100}% → 5000%. Any schematic created via ZERO_NEW (which embeds this legend) will have its legend rendered far outside the viewport.

   The v6.ts internal ZERO_LEGEND correctly uses units: { x: "px", y: "px" } to anchor the legend at 50 px. The same units must be set here so the two defaults remain consistent.

_{Reviews (10): Last reviewed commit: "SY-3833: Set remoteCreated on navigateTo..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 39.95816% with 287 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.10%. Comparing base (46b6a8b) to head (582d9ce).

Files with missing lines	Patch %	Lines
core/pkg/service/schematic/migrate.go	0.00%	95 Missing ⚠️
core/pkg/service/schematic/codec.gen.go	53.37%	34 Missing and 35 partials ⚠️
oracle/plugin/go/pb/pb.go	25.42%	44 Missing ⚠️
x/go/spatial/codec.gen.go	64.17%	12 Missing and 12 partials ⚠️
oracle/plugin/go/marshal/test_gen.go	0.00%	18 Missing and 1 partial ⚠️
console/src/schematic/selectors.ts	35.71%	5 Missing and 4 partials ⚠️
oracle/plugin/go/marshal/encoder.go	0.00%	8 Missing and 1 partial ⚠️
core/pkg/service/schematic/migrate_auto.gen.go	0.00%	6 Missing ⚠️
console/src/schematic/slice.ts	0.00%	3 Missing and 2 partials ⚠️
console/src/schematic/Schematic.tsx	0.00%	2 Missing ⚠️
... and 5 more

Additional details and impacted files

@@                                         Coverage Diff                                          @@
##           sy-4140-schematic-type-re-organization-and-diagram-optimizations    #2283      +/-   ##
====================================================================================================
- Coverage                                                             64.20%   64.10%   -0.11%     
====================================================================================================
  Files                                                                  2187     2200      +13     
  Lines                                                                110644   111735    +1091     
  Branches                                                               8357     8360       +3     
====================================================================================================
+ Hits                                                                  71042    71628     +586     
- Misses                                                                33497    33909     +412     
- Partials                                                               6105     6198      +93     

Flag	Coverage Δ
alamos-go	55.25% <ø> (ø)
arc-go	76.93% <ø> (ø)
aspen	67.77% <ø> (-0.48%)	⬇️
cesium	82.34% <ø> (-0.05%)	⬇️
client-py	85.91% <ø> (ø)
client-ts	90.22% <100.00%> (+0.03%)	⬆️
console	20.78% <51.16%> (+0.03%)	⬆️
core	67.99% <35.60%> (-0.61%)	⬇️
freighter-go	62.75% <ø> (-0.25%)	⬇️
freighter-integration	1.51% <ø> (ø)
freighter-py	79.96% <ø> (ø)
oracle	62.69% <17.24%> (+0.03%)	⬆️
pluto	55.26% <ø> (-0.07%)	⬇️
x-go	81.68% <64.17%> (-0.14%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.