Skip to content

Commit 893f622

Browse files
josephschorrjosephschorr
authored
feat: add unified schema storage with ReadStoredSchema/WriteStoredSchema (#2924)
Introduce the foundation for unified schema storage, where the entire compiled schema (definitions + schema text + hash) is stored as a single serialized proto blob, chunked across rows for SQL datastores. Datastore interface changes: - Add ReadStoredSchema(ctx, SchemaHash) to Reader interface - Add WriteStoredSchema(ctx, *StoredSchema) to ReadWriteTransaction - Add SchemaHash type with sentinel values for cache bypass DataLayer interface changes: - SnapshotReader now takes (Revision, SchemaHash) to thread cache keys - OptimizedRevision/HeadRevision return SchemaHash alongside Revision - Add SchemaMode configuration for migration path (legacy → dual → new) - Add storedSchemaReaderAdapter for reading from StoredSchema protos - Add writeSchemaViaStoredSchema for building and writing StoredSchema Storage implementation: - Add SQLByteChunker generic chunked blob storage for SQL datastores - Add SQLSingleStoreSchemaReaderWriter for read/write of StoredSchema - Add per-datastore ReadStoredSchema/WriteStoredSchema implementations for postgres, crdb, mysql, spanner, and memdb - Add schema table migrations for all SQL datastores - Add populate migrations to backfill from legacy namespace/caveat tables - Rename MySQL schema table to stored_schema (schema is a reserved word) Caching: - Add SchemaHashCache with LRU + singleflight for schema-by-hash lookups Proxy/middleware updates: - Add ReadStoredSchema/WriteStoredSchema pass-through to all datastore proxies (observable, counting, readonly, singleflight, indexcheck, strictreplicated, checkingreplicated, relationshipintegrity, hashcache) - Update consistency middleware for new HeadRevision/OptimizedRevision signatures Proto changes: - Add StoredSchema message to core.proto with V1StoredSchema containing schema_text, schema_hash, namespace_definitions, caveat_definitions
1 parent 5979cb2 commit 893f622

218 files changed

Lines changed: 10656 additions & 1574 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
7878
- feat(query planner): add recursive direction strategies, and fix IS BFS (https://github.com/authzed/spicedb/pull/2891)
7979
- feat(query planner): introduce query plan outlines and canonicalization (https://github.com/authzed/spicedb/pull/2901)
8080
- Schema v2: introduces support for PostOrder traversal in walk.go (https://github.com/authzed/spicedb/pull/2761) and improve PostOrder walker cycle detection (https://github.com/authzed/spicedb/pull/2902)
81+
- Experimental: Add unified schema storage with ReadStoredSchema/WriteStoredSchema for improved schema read performance (https://github.com/authzed/spicedb/pull/2924)
82+
83+
This feature stores the entire schema as a single serialized proto rather than reading individual namespace and caveat definitions separately, significantly improving schema read performance.
84+
85+
Migration to unified schema storage is controlled by the `--experimental-schema-mode` flag, which supports a 4-phase rolling migration:
86+
87+
1. `read-legacy-write-legacy` (default) - No change; reads and writes use legacy per-definition storage.
88+
2. `read-legacy-write-both` - Reads from legacy storage, writes to both legacy and unified storage. This is the first migration step and backfills the unified schema table.
89+
3. `read-new-write-both` - Reads from unified storage, writes to both. Validates the new read path while maintaining backward compatibility.
90+
4. `read-new-write-new` - Reads and writes only unified storage. This is the final migration target.
91+
92+
**With the SpiceDB Operator:** Configure the operator to roll through stages 1 through 4 in sequence. The operator handles the rolling update of SpiceDB instances at each stage.
93+
94+
**Without the operator:** Progress through the stages manually by updating the `--experimental-schema-mode` flag and performing a rolling restart at each stage. You can also take the system down briefly and move directly from stage 1 to stage 4, which runs the full migration in one step.
8195

8296
### Changed
8397
- Begin deprecation of library "github.com/dlmiddlecote/sqlstats" (https://github.com/authzed/spicedb/pull/2904).

‎docs/spicedb.md‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎internal/caveats/run_test.go‎

Lines changed: 39 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -454,7 +454,7 @@ func TestRunCaveatExpressions(t *testing.T) {
454454
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
455455
req.NoError(err)
456456

457-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
457+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
458458
caveat firstCaveat(first int) {
459459
first == 42
460460
}
@@ -466,12 +466,12 @@ func TestRunCaveatExpressions(t *testing.T) {
466466
caveat thirdCaveat(third bool) {
467467
third
468468
}
469-
`, nil, req)
470-
headRevision, err := ds.HeadRevision(t.Context())
469+
`, nil)
470+
headRevisionResult, err := ds.HeadRevision(t.Context())
471471
req.NoError(err)
472472

473473
dl := datalayer.NewDataLayer(ds)
474-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
474+
sr, err := dl.SnapshotReader(headRevisionResult.Revision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
475475
req.NoError(err)
476476

477477
for _, debugOption := range []RunCaveatExpressionDebugOption{
@@ -514,17 +514,18 @@ func TestRunCaveatWithMissingMap(t *testing.T) {
514514
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
515515
req.NoError(err)
516516

517-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
517+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
518518
caveat some_caveat(themap map<any>) {
519519
themap.first == 42
520520
}
521-
`, nil, req)
521+
`, nil)
522522

523-
headRevision, err := ds.HeadRevision(t.Context())
523+
headRevisionResult, err := ds.HeadRevision(t.Context())
524524
req.NoError(err)
525+
headRevision := headRevisionResult.Revision
525526

526527
dl := datalayer.NewDataLayer(ds)
527-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
528+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
528529
req.NoError(err)
529530

530531
result, err := RunSingleCaveatExpression(
@@ -546,17 +547,18 @@ func TestRunCaveatWithEmptyMap(t *testing.T) {
546547
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
547548
req.NoError(err)
548549

549-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
550+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
550551
caveat some_caveat(themap map<any>) {
551552
themap.first == 42
552553
}
553-
`, nil, req)
554+
`, nil)
554555

555-
headRevision, err := ds.HeadRevision(t.Context())
556+
headRevisionResult, err := ds.HeadRevision(t.Context())
556557
req.NoError(err)
558+
headRevision := headRevisionResult.Revision
557559

558560
dl := datalayer.NewDataLayer(ds)
559-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
561+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
560562
req.NoError(err)
561563

562564
_, err = RunSingleCaveatExpression(
@@ -580,21 +582,22 @@ func TestRunCaveatMultipleTimes(t *testing.T) {
580582
rawDS, err := memdb.NewMemdbDatastore(0, 0, memdb.DisableGC)
581583
req.NoError(err)
582584

583-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
585+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
584586
caveat some_caveat(themap map<any>) {
585587
themap.first == 42
586588
}
587589
588590
caveat another_caveat(somecondition int) {
589591
somecondition == 42
590592
}
591-
`, nil, req)
593+
`, nil)
592594

593-
headRevision, err := ds.HeadRevision(t.Context())
595+
headRevisionResult, err := ds.HeadRevision(t.Context())
594596
req.NoError(err)
597+
headRevision := headRevisionResult.Revision
595598

596599
dl := datalayer.NewDataLayer(ds)
597-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
600+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
598601
req.NoError(err)
599602

600603
runner := NewCaveatRunner(types.Default.TypeSet)
@@ -652,17 +655,18 @@ func TestRunCaveatWithMissingDefinition(t *testing.T) {
652655
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
653656
req.NoError(err)
654657

655-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
658+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
656659
caveat existing_caveat(param int) {
657660
param == 42
658661
}
659-
`, nil, req)
662+
`, nil)
660663

661-
headRevision, err := ds.HeadRevision(t.Context())
664+
headRevisionResult, err := ds.HeadRevision(t.Context())
662665
req.NoError(err)
666+
headRevision := headRevisionResult.Revision
663667

664668
dl := datalayer.NewDataLayer(ds)
665-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
669+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
666670
req.NoError(err)
667671

668672
// Try to run a caveat that doesn't exist
@@ -684,20 +688,21 @@ func TestCaveatRunnerPopulateCaveatDefinitionsForExpr(t *testing.T) {
684688
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
685689
req.NoError(err)
686690

687-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
691+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
688692
caveat first_caveat(firstparam int) {
689693
firstparam == 42
690694
}
691695
caveat second_caveat(secondparam string) {
692696
secondparam == "hello"
693697
}
694-
`, nil, req)
698+
`, nil)
695699

696-
headRevision, err := ds.HeadRevision(t.Context())
700+
headRevisionResult, err := ds.HeadRevision(t.Context())
697701
req.NoError(err)
702+
headRevision := headRevisionResult.Revision
698703

699704
dl := datalayer.NewDataLayer(ds)
700-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
705+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
701706
req.NoError(err)
702707

703708
runner := NewCaveatRunner(types.Default.TypeSet)
@@ -732,17 +737,18 @@ func TestCaveatRunnerEmptyExpression(t *testing.T) {
732737
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
733738
req.NoError(err)
734739

735-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
740+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
736741
caveat test_caveat(param int) {
737742
param == 42
738743
}
739-
`, nil, req)
744+
`, nil)
740745

741-
headRevision, err := ds.HeadRevision(t.Context())
746+
headRevisionResult, err := ds.HeadRevision(t.Context())
742747
req.NoError(err)
748+
headRevision := headRevisionResult.Revision
743749

744750
dl := datalayer.NewDataLayer(ds)
745-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
751+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
746752
req.NoError(err)
747753

748754
runner := NewCaveatRunner(types.Default.TypeSet)
@@ -813,17 +819,18 @@ func TestUnknownCaveatOperation(t *testing.T) {
813819
rawDS, err := dsfortesting.NewMemDBDatastoreForTesting(t, 0, 0, memdb.DisableGC)
814820
req.NoError(err)
815821

816-
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(rawDS, `
822+
ds, _ := testfixtures.DatastoreFromSchemaAndTestRelationships(t, rawDS, `
817823
caveat test_caveat(param int) {
818824
param == 42
819825
}
820-
`, nil, req)
826+
`, nil)
821827

822-
headRevision, err := ds.HeadRevision(t.Context())
828+
headRevisionResult, err := ds.HeadRevision(t.Context())
823829
req.NoError(err)
830+
headRevision := headRevisionResult.Revision
824831

825832
dl := datalayer.NewDataLayer(ds)
826-
sr, err := dl.SnapshotReader(headRevision).ReadSchema(t.Context())
833+
sr, err := dl.SnapshotReader(headRevision, datalayer.NoSchemaHashForTesting).ReadSchema(t.Context())
827834
req.NoError(err)
828835

829836
runner := NewCaveatRunner(types.Default.TypeSet)

‎internal/datastore/benchmark/driver_bench_test.go‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ func BenchmarkDatastoreDriver(b *testing.B) {
7777
ctx := b.Context()
7878

7979
// Write the standard schema
80-
ds, _ = testfixtures.StandardDatastoreWithSchema(ds, require.New(b))
80+
ds, _ = testfixtures.StandardDatastoreWithSchema(b, ds)
8181

8282
// Write a fair amount of data, much more than a functional test
8383
for docNum := range numDocuments {
@@ -95,8 +95,9 @@ func BenchmarkDatastoreDriver(b *testing.B) {
9595
// Sleep to give the datastore time to stabilize after all the writes
9696
time.Sleep(1 * time.Second)
9797

98-
headRev, err := ds.HeadRevision(ctx)
98+
headRevResult, err := ds.HeadRevision(ctx)
9999
require.NoError(b, err)
100+
headRev := headRevResult.Revision
100101

101102
b.StartTimer()
102103

0 commit comments

Comments
 (0)