Skip to content

feat: add offline DuckLake service wiring - #488

Merged
weicao merged 2 commits into
mainfrom
task71-ducklake-rebase-20260828
Aug 27, 2026
Merged

feat: add offline DuckLake service wiring#488
weicao merged 2 commits into
mainfrom
task71-ducklake-rebase-20260828

Conversation

@weicao

@weicao weicao commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Task #71: offline DuckLake/httpfs service wiring, rebased onto current main.

Exact identity:

  • head: 9b9a5ef (GitHub verified/valid, web-flow)
  • parent: 2eb4143
  • tree: 5543e845f3f8e6a44f178ec0d5fed3d146e8c772
  • relative scope: 29 paths, no go.mod/go.sum, replication, release, tag, or image changes

The head preserves the reviewed default-off behavior, service-only S3 credential injection, connection lifecycle coverage, and fixed offline extension hashes. Existing parent-baseline connection-refused harness failures are unchanged; focused gates and compile-only checks passed.

This PR is intentionally separate from task #75 table-selection work. Do not publish or alter v0.2.0/latest from this change without the project release decision.

Known limitation

DuckDB 1.5.5 CREATE SECRET is database-global. With lake enabled, a replica connection can observe the service secret even though replica origin is zero-inject. Default-off does not create secrets and does not change current main behavior, so this does not block merging as default-off.

Until replica isolation lands in a later 0.3 follow-up: do not default-on this feature, do not add it to recommended config, and do not enable lake in environments that have replica consumers.

Task #71 reviewed DuckLake service wiring candidate, rebased onto current main.\n\nAssembled through a temporary ref so the final commit is platform-signed; no task #75 changes.
@weicao

weicao commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

Known limitation (CTO, 2026-08-28): DuckDB 1.5.5 CREATE SECRET is database-global. With lake enabled, a replica connection can observe the service secret even though replica origin is zero-inject.

This does not block merging this PR as default-off: default-off does not create secrets and does not change current main behavior.

Until replica isolation lands in a later 0.3 follow-up:

  • do not default-on this feature
  • do not add it to recommended config
  • do not enable lake in environments that have replica consumers

Do not retouch this head for that limitation. After merge, open a follow-up issue and assign it.

@weicao

weicao commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

Cannot merge this head (9b9a5ef3).

Hosted copy-instance and Docker replica mysql/mariadb are candidate-only reds vs parent 2eb4143b: CREATE DATABASE /*!32312 IF NOT EXISTS*/ ... is rejected as service-managed. Parent those jobs are green. Replacement head must unwrap MySQL versioned comments and scan the inner SQL; ordinary CREATE DATABASE must pass.

CTO also requires rejecting CREATE PERSISTENT SECRET (and CREATE OR REPLACE PERSISTENT SECRET) on this PR before merge. One-word scanner fix; no settings/table-function sweep.

Replica-visible database-global secrets remain a known limitation and do not block default-off merge.

Need a new platform-signed head on the same parent, independent review of that exact OID, and no new reds vs parent.

Task #71 follow-up: allow hosted MySQL executable comments that carry ordinary CREATE DATABASE clauses, while rejecting persistent secret creation. This replacement changes only catalog/sensitive_sql.go and is based on the existing signed PR head.
@weicao

weicao commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up replacement head for the two pre-merge lexer fixes:

  • 3e3be6777d290e78d6b38a0eeff118ccaa8c7f7a (parent 9b9a5ef3dae0a2794375e654be4b5f0ad441dd2b, tree cdfcb7dc831b07a27136ce6cc026dd94d49293dd)
  • GitHub commit verification: verified=true, reason=valid, committer GitHub <noreply@github.com>
  • Relative to the prior PR head, only catalog/sensitive_sql.go changed.

The scanner now parses MySQL /*!...*/ contents so hosted CREATE DATABASE /*!32312 IF NOT EXISTS*/ ... (including charset/encryption clauses) is allowed, while executable comments containing sensitive SQL remain rejected. CREATE PERSISTENT SECRET and CREATE OR REPLACE PERSISTENT SECRET are rejected.

Exact-head checks passed: catalog sensitive boundary, tagged backend sensitive/parser tests, tagged pgserver compile, and a disposable executable-comment probe. PR remains open for independent review.

@weicao
weicao force-pushed the task71-ducklake-rebase-20260828 branch from 27c9698 to 3e3be67 Compare August 27, 2026 20:43

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent approval for exact head 3e3be67 (parent 9b9a5ef, tree cdfcb7dc831b07a27136ce6cc026dd94d49293dd). Reviewed task #80 evidence: FlightSQL concurrent/eviction/prepared-transaction checks exit 0 (log 15be53664c318bd915cb9ba98ab6551a0039b51c8d706a76827cb2aa435bfcb5), configuration default-off/redaction exit 0 (log 4b93845f8f5d23b60fa27889cb8a2b7c17c4bad67275d7f038734ece6fc3abca), and hosted backend audit-redaction pass; the local ICU failure is environment-only. Approval is limited to this exact OID and the agreed scope.

@weicao
weicao merged commit dd6d791 into main Aug 27, 2026
25 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants