adapter: bootstrap storage collections in dependency order#35933
Open
teskje wants to merge 1 commit intoMaterializeInc:mainfrom
Open
adapter: bootstrap storage collections in dependency order#35933teskje wants to merge 1 commit intoMaterializeInc:mainfrom
teskje wants to merge 1 commit intoMaterializeInc:mainfrom
Conversation
Contributor
|
Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone. PR title guidelines
Pre-merge checklist
|
teskje
force-pushed
the
bootstrap-storage-topological
branch
from
9779e6f to
be78412
Compare
New builtin MVs that depend on other builtin MVs could fail as-of selection because the bootstrap code registered all derived collections (MVs, CTs) in a single batch. When computing a derived collection's since, the code queried the storage controller for dependency frontiers, but dependencies in the same batch weren't registered yet, so their (persisted) sinces were missed. This led to sinces that were too low, causing unsatisfiable as-of constraints. Fix by registering collections in topological layers: each iteration drains and registers only the collections whose storage dependencies have all been registered. This ensures dependency frontiers are available when bumping the sinces of derived collections.
teskje
force-pushed
the
bootstrap-storage-topological
branch
from
be78412 to
8e77a62
Compare
Contributor
Author
|
All the nightly failures look unrelated, so this should be good to go. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New builtin MVs that depend on other builtin MVs could fail as-of selection because the bootstrap code registered all derived collections (MVs, CTs) in a single batch. When computing a derived collection's since, the code queried the storage controller for dependency frontiers, but dependencies in the same batch weren't registered yet, so their (persisted) sinces were missed. This led to sinces that were too low, causing unsatisfiable as-of constraints.
Fix by registering collections in topological layers: each iteration drains and registers only the collections whose storage dependencies have all been registered. This ensures dependency frontiers are available when bumping the sinces of derived collections.
Motivation
Fixes MaterializeInc/database-issues#11273
Verification
Triggered the 0dt smoke test a couple times, which reproduced the issue before.