Skip to content

fix: recover persistent catalog WAL after restart - #481

Merged
weicao merged 1 commit into
mainfrom
fix/wal-replay-default-db
Aug 24, 2026
Merged

fix: recover persistent catalog WAL after restart#481
weicao merged 1 commit into
mainfrom
fix/wal-replay-default-db

Conversation

@weicao

@weicao weicao commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Root cause

DuckDB 1.5.5 replays COMMENT ON COLUMN as ReplayAlter -> SetColumnComment. When the table contains a generated column, replay rebinds the generated expression before the main database has been installed as DuckDB's default database, aborting startup with:

Calling DatabaseManager::GetDefaultDatabase with no default database set

The trigger matrix is specifically generated column + column comment; ordinary tables and generated tables without column comments reopen successfully.

Fix

  • Initialize an in-memory DuckDB default catalog first.
  • Register MyDuck scalar UDFs before persistent WAL replay.
  • Attach and USE the configured persistent catalog after the default database exists.
  • Initialize every later connection on the configured persistent catalog.
  • Preserve Restart(true) with attach-level READ_ONLY; in-memory restart retains the prior DSN behavior.

This avoids a DuckDB dependency or Doltgres migration and is compatible with generated expressions that reference MyDuck UDFs.

Regression coverage

  • ordinary/generated x with/without column comment matrix
  • mysql_rand generated expression and BLOB-backed VECTOR storage shape
  • file-position and GTID replication metadata
  • schema, comment, generated data, vector bytes, and position snapshots before/after replay
  • real subprocess exit without provider/DuckDB shutdown, followed by process reopen
  • eight concurrent connections selecting the persistent default catalog
  • two isolated providers with the same catalog name and no user tables in the in-memory bootstrap catalog
  • read-only restart rejects writes and writable restart restores them

Verification

  • go test -race -tags duckdb_arrow ./catalog -run '^(TestPersistentCatalogWALReplay|TestPersistentCatalogWALReplayAcrossProcess|TestPersistentCatalogProvidersAreIsolated|TestDatabaseProviderRestartClosesConnectionsBeforeReopen)$' -count=1
  • GOFLAGS=-tags=duckdb_arrow go test ./catalog -count=1
  • GOFLAGS=-tags=duckdb_arrow go test ./... -run '^$' -count=1
  • GOFLAGS=-tags=duckdb_arrow go vet ./catalog
  • focused generated/catalog/persistence engine tests pass

Candidate snapshot

  • exact parent: 749b6164b105dafc08d627fa7ab522f4805c654b
  • signed head: a56af4dccce0d40e2b84157a60026098dbfadbc8
  • tree: 30453d90c23f446354ffa77a84dc9dd8b8e94646
  • commit verification: GitHub verified, signed by GitHub
  • files: catalog/provider.go, catalog/provider_restart_test.go, catalog/provider_wal_replay_test.go
  • go.mod / go.sum: unchanged

Do not merge until independent review completes. The Doltgres v1.2.0 candidate remains separately blocked from merge or release until this fix is reviewed.

Initialize DuckDB with an in-memory default catalog, register MyDuck UDFs, and then attach the persistent catalog so WAL replay can bind generated expressions safely. Keep new sessions on the configured catalog and preserve read-only restart behavior.

Cover real process restart, ordinary and generated comment matrices, BLOB vector storage, replication positions, snapshots, concurrency, and provider isolation.

Generated through the GitHub platform commit API.

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved exact signed head a56af4dccce0d40e2b84157a60026098dbfadbc8 only.

Local candidate validation passed: focused catalog restart/WAL replay (in-process and subprocess), provider isolation, and mysql_rand UDF registration stability; race run passed, focused tests passed, and diff-check passed. Scope is limited to catalog/provider.go, catalog/provider_restart_test.go, and catalog/provider_wal_replay_test.go; parent is exact 749b6164b105dafc08d627fa7ab522f4805c654b.

CI attempt 2 packages and Binlog Group 1 (including ServerRestart) passed. Go Query Engine failures match exact-parent run 32628342514/job 97167117220; Binlog Group 2 TestAutoRestartReplica matches exact-parent 32628342547/job 97167117056 (0 -> 1105, unsupported row-event flags 10). Attempt-1 CreateCatalog startup and AutoReconnect timing failures are non-reproducing/exact-base harness behavior per attached waiver. No product blocker found in this PR scope; do not merge or publish from this review.

weicao added a commit that referenced this pull request Aug 24, 2026
Review-only migration candidate. Depends on task #54-approved PR #481, which remains unmerged; do not merge or publish independently.
weicao added a commit that referenced this pull request Aug 24, 2026
Review-only migration candidate. Fixes the Python autocommit transaction lifetime regression found in task #55. Depends on task #54-approved PR #481, which remains unmerged; do not merge or publish independently.
@weicao
weicao merged commit 3ec2b33 into main Aug 24, 2026
11 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants