BE-635: Replace composite-type inserts to support zero-downtime additive migrations#8959
Open
TimDiekmann wants to merge 5 commits into
Open
Conversation
…ameters Composite row types on the wire encode the table's column count, so any additive migration broke inserts from binaries compiled against the old schema during rollout. Live write paths now name every inserted column explicitly and send one array parameter per column via unnest, letting old binaries coexist with an additively migrated schema. - rework PostgresRow to produce column-parameter pairs consumed by InsertStatement::compile_rows, deriving statement and parameters from a single sequence - declare storage types per insertable column (InsertableColumn) instead of deriving them from filter parameter types - rewrite the embedding upserts to scalar/array parameters and drop the local composite row structs - verify affected row counts of the bulk inserts - name inserted columns in the positional entity_is_of_type inserts Snapshot restore still uses composite types and is left for a follow-up.
Replaces the 32 composite-type UNNEST inserts in the snapshot batch writers with the InsertStatement machinery introduced for the live write paths, removing the last composite row types on the wire and all composite ToSql derives. - extend InsertStatementOptions with a table-name override, DISTINCT, and an extensible OnConflict enum; the tmp-table staging convention moves into the snapshot's insert_rows_batch helper - PostgresRow names its table via fn table() -> TableName, allowing tables outside the Table enum - DatabaseColumn now carries the physical column facts with &self receivers and no Copy bound; parameter_type moves to the new FilterColumn trait; the unused nullable method and eight unused row structs are removed - introduce PostgresType as the single storage-type authority with canonical Postgres type names, replacing the ad-hoc cast enum - add column enums, PostgresRow impls, Table variants, and the principal_type/policy_effect types for all snapshot tables Validated with the crate test suite, the hash-backend-integration restore roundtrip, the graph HTTP suite, and manual dump/restore equivalence checks. The latter surfaced a pre-existing bug where the restore re-applies data type conversions to canonical property values (BE-634).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
PR SummaryHigh Risk Overview
Live write paths (entity bulk insert, data-type conversion inserts) switch from
Reviewed by Cursor Bugbot for commit e67d406. Bugbot is set up for automated code reviews on this repo. Configure here. |
github-actions
Bot
added
area/libs
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Postgres write/query layer (and snapshot restore paths) to avoid composite row types on the wire by generating INSERT ... (col, ...) SELECT * FROM UNNEST($1::<type>[], ...) statements, enabling zero-downtime additive schema migrations where old binaries can keep inserting without referencing newly added nullable/defaulted columns.
Changes:
- Replaced composite-type bulk inserts with explicit column lists + parallel array parameters via
UNNEST, with configurableDISTINCT/ON CONFLICTbehavior. - Centralized storage-level SQL type rendering in a new
PostgresType(built-in names sourced frompostgres_types::Type) and updated HashQL filter SQL snapshots accordingly. - Refactored snapshot restore batch insert code to use a common helper that targets
<table>_tmpstaging tables and uses the new insert statement compilation.
Reviewed changes
Copilot reviewed 79 out of 79 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/@local/hashql/eval/tests/ui/postgres/tuple-construction.stdout | Updated expected SQL output for new ::bool casts / formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/struct-construction.stdout | Updated expected SQL output for new ::bool casts / formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/opaque-passthrough.stdout | Updated expected SQL output for new ::bool casts / formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/nested-if-input-branches.stdout | Updated expected SQL output for ::int4 / ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/multiple-filters.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/mixed-sources-filter.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/minimal-select-no-extra-joins.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/logical-and-inputs.stdout | Updated expected SQL output for ::int4 / ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/list-construction.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/let-binding-propagation.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/input-parameter-load.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/input-parameter-exists.stdout | Updated expected SQL output for ::int4 / ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/if-input-branches.stdout | Updated expected SQL output for ::int4 / ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/unary_not.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/unary_neg.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/unary_bitnot.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/temporal_decision_time_interval.snap | Updated expected SQL output for int4[] array literal typing. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/switch_int_many_branches.snap | Updated expected SQL output for ::int4 / ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/straight_line_goto_chain.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/property_mask.snap | Updated expected SQL output for int4[] array literal typing. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/property_field_equality.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/nested_property_access.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/left_entity_filter.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/island_exit_with_live_out.snap | Updated expected SQL output for int4[] array literal typing. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/island_exit_switch_int.snap | Updated expected SQL output for ::int4 / ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/island_exit_goto.snap | Updated expected SQL output for int4[] array literal typing. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/island_exit_empty_arrays.snap | Updated expected SQL output for int4[] empty array typing. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/field_index_projection.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/field_by_name_projection.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/dynamic_index_projection.snap | Updated expected SQL output for ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/diamond_cfg_merge.snap | Updated expected SQL output for ::int4 / ::bool casts. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/binary_sub_numeric_cast.snap | Updated expected SQL output for ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/binary_bitor_boolean_or.snap | Updated expected SQL output for ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/binary_bitor_bigint_cast.snap | Updated expected SQL output for ::int8 / ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/binary_bitand_boolean_and.snap | Updated expected SQL output for ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/binary_bitand_bigint_cast.snap | Updated expected SQL output for ::int8 / ::bool casts and formatting. |
| libs/@local/hashql/eval/tests/ui/postgres/env-captured-variable.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/entity-web-id-equality.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/entity-uuid-equality.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/entity-type-ids-lateral.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/entity-archived-check.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/dict-construction.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/constant-true-filter.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/postgres/comparison-no-cast.stdout | Updated expected SQL output for ::bool casts and line numbering. |
| libs/@local/hashql/eval/tests/ui/graph/read/entity/field-access-tuple.stdout | Updated expected debug formatting for decimal significands. |
| libs/@local/hashql/eval/tests/ui/graph/read/entity/field-access-struct.stdout | Updated expected debug formatting for decimal significands. |
| libs/@local/hashql/eval/tests/ui/graph/read/entity/arithmetic-comparisons-or.stdout | Updated expected debug formatting for decimal significands. |
| libs/@local/hashql/eval/tests/ui/graph/read/entity/arithmetic-comparisons-and.stdout | Updated expected debug formatting for decimal significands. |
| libs/@local/hashql/eval/src/postgres/parameters.rs | Updated parameter kind → Postgres type mapping (Int8, Bool, TstzRange). |
| libs/@local/hashql/eval/src/postgres/filter/tests.rs | Updated test documentation strings for new cast spellings (int8, bool). |
| libs/@local/hashql/eval/src/postgres/filter/mod.rs | Updated filter SQL generation to use Bool/Int4/Int8 types. |
| libs/@local/graph/postgres-store/src/store/postgres/query/table.rs | Refactored column traits and added many table/column storage type definitions. |
| libs/@local/graph/postgres-store/src/store/postgres/query/statement/mod.rs | Re-exported new insert statement types. |
| libs/@local/graph/postgres-store/src/store/postgres/query/statement/insert.rs | Replaced builder-based insert construction with UNNEST-based bulk insert compilation. |
| libs/@local/graph/postgres-store/src/store/postgres/query/rows.rs | Replaced composite-row ToSql derives with columnar parameter generation per table. |
| libs/@local/graph/postgres-store/src/store/postgres/query/postgres_type.rs | Added storage-level Postgres type enum and transpilation logic. |
| libs/@local/graph/postgres-store/src/store/postgres/query/mod.rs | Wired in new postgres_type module and updated exports. |
| libs/@local/graph/postgres-store/src/store/postgres/query/expression/table_reference.rs | Added TableName::as_str() helper used for staging table naming. |
| libs/@local/graph/postgres-store/src/store/postgres/query/expression/mod.rs | Removed PostgresType export from expression module (now centralized). |
| libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs | Removed old PostgresType definition and row-expansion expression support. |
| libs/@local/graph/postgres-store/src/store/postgres/query/compile.rs | Updated compiler to use new PostgresType and expanded unsupported table list for embeddings. |
| libs/@local/graph/postgres-store/src/store/postgres/ontology/read.rs | Adjusted string handling for column names (to_owned()). |
| libs/@local/graph/postgres-store/src/store/postgres/ontology/property_type.rs | Replaced composite UNNEST($1::<composite>[] ) embedding insert with explicit columns/params. |
| libs/@local/graph/postgres-store/src/store/postgres/ontology/entity_type.rs | Replaced composite UNNEST($1::<composite>[] ) embedding insert with explicit columns/params. |
| libs/@local/graph/postgres-store/src/store/postgres/ontology/data_type.rs | Replaced composite inserts with new InsertStatement + execute_raw and row-count checks; updated embedding insert. |
| libs/@local/graph/postgres-store/src/store/postgres/knowledge/entity/mod.rs | Switched multiple inserts to InsertStatement + execute_raw; updated entity embeddings insert to columnar unnest. |
| libs/@local/graph/postgres-store/src/snapshot/principal/table.rs | Replaced composite row ToSql derives with PostgresRow + per-column metadata for snapshot restore. |
| libs/@local/graph/postgres-store/src/snapshot/principal/channel.rs | Updated comment to match new sender structure. |
| libs/@local/graph/postgres-store/src/snapshot/principal/batch.rs | Migrated principal snapshot batch inserts to shared insert_rows_batch helper. |
| libs/@local/graph/postgres-store/src/snapshot/policy/table.rs | Replaced composite row ToSql derives with PostgresRow + per-column metadata for snapshot restore. |
| libs/@local/graph/postgres-store/src/snapshot/policy/batch.rs | Migrated policy snapshot batch inserts to shared insert_rows_batch helper. |
| libs/@local/graph/postgres-store/src/snapshot/ontology/property_type/batch.rs | Migrated ontology property-type snapshot batch inserts to shared helper + new conflict options. |
| libs/@local/graph/postgres-store/src/snapshot/ontology/metadata/batch.rs | Migrated ontology metadata snapshot batch inserts to shared helper + new conflict options. |
| libs/@local/graph/postgres-store/src/snapshot/ontology/entity_type/batch.rs | Migrated ontology entity-type snapshot batch inserts to shared helper + new conflict options. |
| libs/@local/graph/postgres-store/src/snapshot/ontology/data_type/batch.rs | Migrated ontology data-type snapshot batch inserts to shared helper + new conflict options. |
| libs/@local/graph/postgres-store/src/snapshot/mod.rs | Added SnapshotInsertOptions and insert_rows_batch helper targeting <table>_tmp tables. |
| libs/@local/graph/postgres-store/src/snapshot/entity/batch.rs | Migrated entity snapshot batch inserts to shared helper + new conflict options. |
| libs/@local/graph/postgres-store/src/snapshot/action/table.rs | Replaced composite row ToSql derives with PostgresRow + per-column metadata for snapshot restore. |
| libs/@local/graph/postgres-store/src/snapshot/action/batch.rs | Migrated action snapshot batch inserts to shared helper + new conflict options. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8959 +/- ##
==========================================
- Coverage 59.63% 59.37% -0.27%
==========================================
Files 1354 1355 +1
Lines 131373 132272 +899
Branches 6021 6024 +3
==========================================
+ Hits 78344 78535 +191
- Misses 52099 52807 +708
Partials 930 930
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…osite-type-inserts-to-support-zero-downtime # Conflicts: # libs/@local/hashql/eval/tests/ui/graph/read/entity/arithmetic-comparisons-and.stdout # libs/@local/hashql/eval/tests/ui/graph/read/entity/arithmetic-comparisons-or.stdout # libs/@local/hashql/eval/tests/ui/graph/read/entity/field-access-struct.stdout # libs/@local/hashql/eval/tests/ui/graph/read/entity/field-access-tuple.stdout
indietyp
reviewed
Jul 7, 2026
thehabbos007
approved these changes
Jul 7, 2026
thehabbos007
left a comment
thehabbos007
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me. Using unnest should also have slightly better performance in some situations as far as I know. I suggest we also await Bilal's review
indietyp
requested changes
Jul 7, 2026
indietyp
left a comment
indietyp
left a comment
Member
There was a problem hiding this comment.
I'm uncomfortable with the size of the unverified, hand-authored surface here: the 33 columnar_parameters impls are ~950 lines of manual AoS-to-SoA transposition over ~300 hand-written column/type pairs, none of it const-, macro-, or otherwise generated, with virtually 0 coverage reported by codecov over these files. While I appreciate the fact that a future PR may create conformance tests over information schemas, such tests would not catch issues in e.g. columnar_parameters column attribution during transposition.
The benchmarks have also shown a major regression; is this just noise or something more substantial here?
| } | ||
|
|
||
| fn columnar_parameters<'r>( | ||
| rows: &'r [Self], |
Member
There was a problem hiding this comment.
We started using descriptive names for lifetimes, so I reckon this should be 'rows instead.
| pub transaction_time: LeftClosedTemporalInterval<TransactionTime>, | ||
| } | ||
|
|
||
| use crate::store::postgres::query::{ |
Member
There was a problem hiding this comment.
please move the use statement to the top
|
|
||
| /// Transposes `rows` into one array parameter per column. | ||
| /// | ||
| /// Implementations destructure `Self` exhaustively so a new field is a compile error |
Member
There was a problem hiding this comment.
I would promote this section to a proper # Implementation Note instead of just a description. The conversion into columnar_properties may lead to data corruption if not done carefully, such as properties missing or misalignment (as the description itself states). Additionally, a small declarative macro may help reduce repetitive code throughout this change, which should also reduce the potential hazard of a wrong implementation in the future.
| pub role_id: Uuid, | ||
| } | ||
|
|
||
| use crate::store::postgres::query::{ |
Member
There was a problem hiding this comment.
please move these at the top (I will stop to mention them all now, but there are a bunch of others)
Member
There was a problem hiding this comment.
It seems odd to me that we're deferring snapshot tests for this, even though we explicitly add the implementation here. Feels like a potential footgun.
| fn parameter_type(self) -> ParameterType; | ||
| fn nullable(self) -> bool; | ||
| fn as_str(self) -> &'static str; | ||
| /// A physical column of a database table: its SQL name and storage type. |
Member
There was a problem hiding this comment.
That is incorrect; it is also implemented for, e.g. DataTypeEmbeddings::Distance, which is not a physical column.
| fn as_str(self) -> &'static str; | ||
| /// A physical column of a database table: its SQL name and storage type. | ||
| pub trait DatabaseColumn { | ||
| fn as_str(&self) -> &str; |
Member
There was a problem hiding this comment.
Why did this degrade to &str instead of &static str? The idea behind using &'static str was that we could ensure that the value is well... static and has been properly escaped. We cannot do that in this case anymore.
| rows: &'r [R], | ||
| options: InsertStatementOptions, | ||
| ) -> (String, Vec<Box<dyn ToSql + Send + Sync + 'r>>) { | ||
| let (columns, parameters): (Vec<_>, Vec<_>) = |
Member
There was a problem hiding this comment.
We may also want to add a debug assert to ensure that no column is named twice and that the number of columns matches the expected count.
| // TODO: Use a builder which knows `values` at compile time | ||
| let InsertValueItem::Values(values) = &mut self.statement.values else { | ||
| unreachable!() | ||
| pub fn compile_rows_with<'r, R: PostgresRow>( |
Member
There was a problem hiding this comment.
Isn't there a potential footgun here with arrays as arguments and UNNEST? Nothing prevents the user from supplying a PostgresType::Array, which is then unnested.
| let mut web_ids = Vec::with_capacity(rows.len()); | ||
| let mut entity_uuids = Vec::with_capacity(rows.len()); | ||
| let mut draft_ids = Vec::with_capacity(rows.len()); | ||
| let mut propertys = Vec::with_capacity(rows.len()); |
Member
Author
There was a problem hiding this comment.
I should stop using sed 😂
Contributor
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🌟 What is the purpose of this PR?
Schema changes must survive the rollout window (graph-migrate done, old binary still serving). Composite row types on the wire encode the table's column count, so any column change on a composite-inserted table broke every insert from the old binary — rolling migrations were impossible and every schema change required a maintenance window.
This PR replaces all composite-type inserts (live write paths and snapshot restore) with explicit column lists and one array parameter per column via
unnest. An old binary simply doesn't mention a new nullable/defaulted column and keeps working, so additive migrations roll out with zero downtime.🔗 Related links
🔍 What does this change?
PostgresRowproduces column–parameter pairs;InsertStatementderives statement, casts, and parameters from that single sequence, so they cannot fall out of orderInsertStatementOptionswith table-name override,DISTINCT, and an extensibleOnConflictenum; the<table>_tmpstaging convention lives in the snapshot'sinsert_rows_batchhelperToSqlderive removedPostgresTypebecomes the single storage-type authority (built-in names sourced frompostgres_types::Type);DatabaseColumnnow carries the physical column facts, filter typing moves to the newFilterColumntraitnullable(), eight unused row structs)Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
🐾 Next steps
information_schemaconformance test for the hand-written column/type mappingsDatabaseTabletrait linking table → columns → row, separating physical from computed columns🛡 What tests cover this?
hash-graph-postgres-storeintegration suite (277 tests) against live Postgreshash-backend-integrationsnapshot-restore roundtrip (48 tests) and the graph HTTP suitedump → wipe → restore → dumpequivalence check (canonical JSON comparison)❓ How to test this?
cargo nextest run --package hash-graph-postgres-store --all-features(needs the compose Postgres + plainhash-graph-migrations)cd tests/hash-backend-integration && yarn vitest --run src/tests/subgraph(needs the graph with--embed-admin)