FE-1130: Token frame format v2 — packed struct token layout

[kube]

🌟 What is the purpose of this PR?

Replaces the simulation engine's uniform one-Float64-slot-per-dimension token encoding with a schema-driven packed struct per token (format v2), everywhere: interactive engine, Monte Carlo, frame readers, and the encoding playground. Booleans now occupy a single byte instead of eight, places are byte-addressed, and the per-colour layout module gives future wider types (128-bit UUIDs, FE-1121) a place to slot in as u64×2 fields.

🔗 Related links

• FE-1130: Petrinaut: token frame format v2 — packed struct token layout (internal)
• FE-1121: Add UUID discrete type to Petrinaut (internal — the design notes there build on this layout machinery)
• Stacked on FE-1129: Token encoding playground (Storybook) #8943 (playground), which stacks on H-6519: Add discrete token attribute types #8764 (discrete types).

🚫 Blocked by

• H-6519: Add discrete token attribute types #8764
• FE-1129: Token encoding playground (Storybook) #8943 (base branch — merges first)

🔍 What does this change?

• New engine/token-layout.ts — single source of truth for the physical layout: real/integer → f64 (8 B), boolean → u8 (1 B); fields ordered by decreasing alignment (stable), stride rounded up to 8 B so every f64 field stays addressable through a shared Float64Array view (no DataView in hot paths). Read/write goes through readTokenRecord / writeTokenValue / encodeTokenToBytes only.
• EngineFrame bumped to FRAME_VERSION = 2: per-place state is byte-addressed ({ byteOffset, count, strideBytes }), the header records the token region's byte length, snapshots carry Uint8Arrays, and all whole-token moves (kernel-produced insertions, removal/compaction) are byte-range copies.
• Dynamics integrate real fields only: Euler walks realFieldF64Offsets; discrete bytes are copied through untouched (structurally stronger than zeroing derivatives).
• Determinism preserved: kernels sample distributions in element declaration order with the seeded RNG, then pack per layout — same seed, same run.
• Monte Carlo frame buffers use byte capacities (tokenByteCount / tokenByteCapacity, same ×2 growth policy, dual u8/f64 views).
• Public API break: getPlaceTokenValues / SimulationPlaceTokenValues removed — raw f64 access is meaningless under mixed field widths; consumers use getPlaceTokens (typed records). The monorepo was swept: no usage outside the petrinaut packages.
• Playground drops its v1 mode and reuses the core layout module, so the story renders the real engine layout.
• simulation/ARCHITECTURE.md and engine/README.md token-storage sections rewritten for the packed layout.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies an npm-publishable library and I have added a changeset file(s)
  • (token-frame-format-packed-structs.md — dedicated entry for the layout change and the API slimming)

📜 Does this require a change to the docs?

The changes in this PR:

• are internal and do not require a docs change
  • (frame format internals; user-facing behaviour is unchanged — user docs were updated for typed dimensions in H-6519: Add discrete token attribute types #8764)

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Engine throughput has not been benchmarked v1 vs v2. Nothing suggests a regression (element-wise copy loops became bulk byte copies), but there is no measurement.
• Frames are ephemeral per run, so no data migration is needed; the version bump makes any stale-frame decode fail loudly rather than silently misread.

🐾 Next steps

• FE-1121: UUID as a u64×2 field (align 8) with bigint at the JS boundary — the layout machinery here is built for it.
• Possible follow-up: authoring-time warning when a place has dynamics enabled but its colour has no real elements (currently a silent no-op by design).

🛡 What tests cover this?

• New engine/token-layout.test.ts (layout computation, encode/decode round-trips incl. u8 booleans and integer rounding) + a discrete-fields-untouched dynamics integration test
• All engine/Monte Carlo/frame-reader suites updated to the byte-addressed fixtures — 553 tests in @hashintel/petrinaut-core, 137 in @hashintel/petrinaut
• lint:tsc (tsgo) and lint:eslint (oxlint) clean for both packages

❓ How to test this?

1. Checkout the branch, yarn dev in libs/@hashintel/petrinaut.
2. Run a Quick Simulation on a net with typed tokens (include a Boolean dimension); confirm initial state, transitions, dynamics, visualizers, and the timeline behave as before.
3. Run an Experiment (Monte Carlo) with metrics; confirm progress and metric charts.
4. Open Dev / Token Encoding Playground: a Boolean dimension renders as one u8 byte with hatched tail padding, sizeof(token) reflects the packed stride.

📹 Demo

See the token-encoding playground story: the memory view renders the exact v2 layout this PR ships (u8 booleans, alignment padding, byte-addressed slots).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hash	Ready	Preview, Comment	Jul 6, 2026 10:57pm
petrinaut	Ready	Preview, Comment	Jul 6, 2026 10:57pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
hashdotdesign-tokens	Ignored	Preview	Jul 6, 2026 10:57pm

[cursor]

PR Summary

High Risk
This is a breaking internal frame format and public reader API change across stepping, transitions, dynamics, and Monte Carlo—incorrect layout or stride math would corrupt runs, though version checks and broad test updates mitigate that.

Overview
Simulation engine frames move from flat Float64-per-dimension token storage to format v2 (FRAME_VERSION = 2): a Uint8Array token region of packed structs per colour, laid out by new engine/token-layout.ts (f64 for real/integer, u8 for boolean, alignment-ordered fields, 8-byte strides).

Engine & Monte Carlo now use per-place byteOffset / strideBytes, TokenSlotLayout on compiled transitions, byte-block add/remove, and dynamics that integrate only real fields while copying discrete bytes through. Monte Carlo buffers track tokenByteCount / tokenByteCapacity instead of Float64 slot counts.

Public API: removes getPlaceTokenValues and SimulationPlaceTokenValues; getPlaceTokens(place) no longer takes a colour argument; buildMetricState drops the types list. Actual-mode timeline follows the same reader shape.

Tests and docs are updated for the byte layout; token-layout is exported from the core package for the encoding playground.

^{Reviewed by Cursor Bugbot for commit dce97bc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR upgrades Petrinaut’s simulation frame token storage to frame format v2: a packed-struct, byte-addressed token region with a schema-driven per-colour layout (f64 for real/integer, u8 for boolean), and updates the engine, Monte Carlo pipeline, frame readers, and the token encoding playground to use that layout consistently.

Changes:

• Introduces a core TokenSlotLayout abstraction and helpers for computing layouts and reading/writing packed token bytes (computeTokenSlotLayout, readTokenRecord, encodeTokenToBytes, etc.).
• Bumps engine frames to version 2 and converts all token-region handling from Float64Array value slots to Uint8Array byte regions with shared u8/f64 views.
• Updates Monte Carlo buffering/transition effects/dynamics and the dev token encoding playground to reflect the packed layout; removes the now-meaningless raw f64 token-values API.

Reviewed changes

Copilot reviewed 44 out of 45 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/token-memory-view.tsx	Adapts the memory view UI to render TokenSlotLayout fields (byte offsets/sizes and field metadata) from core.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/token-encoding-playground.tsx	Removes v1/v2 toggle and scopes Monaco’s built-in TS language service to the playground lifecycle.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/playground-monaco.ts	Adds enable/disable controls for Monaco TS mode configuration and clears TS markers on unmount.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/physical-layout.ts	Replaces local layout/codec implementation with thin wrappers over core token-layout encode/decode helpers.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/physical-layout.test.ts	Updates playground layout and encode/decode tests to use core TokenSlotLayout.
libs/@hashintel/petrinaut/src/ui/dev/token-encoding-playground/monaco-typescript.d.ts	Extends the local TS defaults typing to include setModeConfiguration.
libs/@hashintel/petrinaut/src/react/playback/provider.test.tsx	Updates playback mocks to match the removal of getPlaceTokenValues.
libs/@hashintel/petrinaut/src/main.ts	Removes the exported SimulationPlaceTokenValues type from the Petrinaut package surface.
libs/@hashintel/petrinaut-core/src/simulation/runtime/simulation.test.ts	Updates frame construction in tests to use an empty Uint8Array token buffer.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/types.ts	Renames Monte Carlo token capacity/count fields to byte-based equivalents.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/transition-effect.ts	Switches transition effects to read/write packed token bytes via token-layout helpers.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/run-state.ts	Converts frame resizing and run summaries to use token byte counts/capacities.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/monte-carlo-simulator.test.ts	Updates Monte Carlo tests to assert byte-sized token regions and to use typed token access.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/internal-types.ts	Changes transition additions to store Uint8Array token blocks rather than number arrays.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-reader.ts	Updates Monte Carlo frame reader to return typed token records from packed bytes.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-operations.ts	Converts dynamics/removal/addition operations to work on packed byte regions.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/frame-buffer.ts	Updates Monte Carlo frame buffer layout to include u8/f64 views over a byte-addressed token region.
libs/@hashintel/petrinaut-core/src/simulation/monte-carlo/advance-run.ts	Updates pending token-additions map type to Uint8Array[].
libs/@hashintel/petrinaut-core/src/simulation/index.ts	Removes SimulationPlaceTokenValues export from the simulation entrypoint.
libs/@hashintel/petrinaut-core/src/simulation/frames/internal-frame.ts	Bumps FRAME_VERSION to 2 and migrates internal frame layout/state to byte offsets + packed token bytes.
libs/@hashintel/petrinaut-core/src/simulation/frames/frame-reader.ts	Updates the main frame reader to decode typed tokens from packed bytes via readTokenRecord.
libs/@hashintel/petrinaut-core/src/simulation/frames/frame-reader.test.ts	Rewrites frame-reader tests to validate copied token records rather than copied f64 buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/types.ts	Updates engine types for byte-addressed dynamics inputs and includes per-place tokenLayout.
libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.ts	Adds the core packed token layout module (layout computation + encode/decode + shared views).
libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.test.ts	Adds unit tests covering layout math, encode/decode round-trips, and alignment checks.
libs/@hashintel/petrinaut-core/src/simulation/engine/token-layout.test-helpers.ts	Adds helpers for constructing/decoding v2 frames and token blocks in tests.
libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.ts	Migrates token removal/compaction to operate on packed byte ranges.
libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.test.ts	Rewrites token-removal tests around packed bytes and typed token decoding helpers.
libs/@hashintel/petrinaut-core/src/simulation/engine/README.md	Updates engine documentation to describe frame format v2 token storage and access patterns.
libs/@hashintel/petrinaut-core/src/simulation/engine/execute-transitions.ts	Updates transition execution to append packed token byte blocks and adjust byte offsets.
libs/@hashintel/petrinaut-core/src/simulation/engine/execute-transitions.test.ts	Updates transition execution tests to validate packed-token behavior via decode helpers.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.ts	Updates possible-transition computation to decode/encode packed token byte blocks and preserve determinism.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.test.ts	Updates possible-transition tests to validate packed byte outputs by decoding token blocks.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-place-next-state.ts	Updates dynamics stepping to integrate real fields only while copying discrete bytes through unchanged.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-place-next-state.test.ts	Updates dynamics tests and adds coverage for “discrete fields untouched” behavior.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-next-frame.ts	Updates frame evolution to apply dynamics on packed bytes and write results back by byte offsets.
libs/@hashintel/petrinaut-core/src/simulation/engine/compute-next-frame.test.ts	Updates next-frame tests to assert on decoded tokens rather than raw numeric buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/check-transition-enablement.test.ts	Updates enablement tests for byte-offset/stride-based place state and v2 frame buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/build-simulation.ts	Updates initial marking packing and dynamics compilation to use packed token layouts and byte buffers.
libs/@hashintel/petrinaut-core/src/simulation/engine/build-simulation.test.ts	Updates build-simulation tests to validate packed layout properties and decoded token records.
libs/@hashintel/petrinaut-core/src/simulation/ARCHITECTURE.md	Updates architecture docs to describe frame format v2 and packed-struct token layout.
libs/@hashintel/petrinaut-core/src/simulation/api.ts	Removes SimulationPlaceTokenValues and getPlaceTokenValues from the public frame reader API.
libs/@hashintel/petrinaut-core/src/index.ts	Exports the new token-layout module API from the package root and removes the old token-values export.
libs/@hashintel/petrinaut-core/src/actual-mode/timeline.ts	Removes getPlaceTokenValues implementation from actual-mode timeline frame reader.
.changeset/h-6519-discrete-token-attribute-types.md	Bumps changeset impact to minor for both packages and notes the v2 frame format change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

libs/@hashintel/petrinaut-core/src/simulation/engine/compute-possible-transition.ts:107

• Typo in comment: "acumulate" → "accumulate".

  // TODO: This should acumulate lambda over time, but for now we just consider that lambda is constant per combination.
  // (just multiply by time since last transition)

libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.ts:62

• This error message still refers to "dimensions"; it should describe the non-zero strideBytes case to match the new frame format terminology.

      if (typeof indices === "number") {
        throw new Error(
          `For place ${placeId} with dimensions, expected Set of token indices to remove, got number.`,
        );
      }

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

libs/@hashintel/petrinaut-core/src/simulation/engine/remove-tokens-from-simulation-frame.ts:62

• Error message still refers to “dimensions”, but the v2 frame format uses strideBytes/byte addressing. This will be misleading when debugging incorrect tokensToRemove shapes for coloured places.

    } else {
      if (typeof indices === "number") {
        throw new Error(
          `For place ${placeId} with dimensions, expected Set of token indices to remove, got number.`,
        );
      }

[Copilot]

Pull request overview

Copilot reviewed 51 out of 51 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 52 out of 52 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 51 out of 51 changed files in this pull request and generated 1 comment.