Skip to content

fix(mariadb): fail closed on unsafe Galera bootstrap#2997

Merged
leon-ape merged 1 commit into
mainfrom
helen/mariadb-galera-bootstrap-fail-closed
Jul 6, 2026
Merged

fix(mariadb): fail closed on unsafe Galera bootstrap#2997
leon-ape merged 1 commit into
mainfrom
helen/mariadb-galera-bootstrap-fail-closed

Conversation

@weicao

@weicao weicao commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • stop pod-0 from rewriting safe_to_bootstrap: 0 to safe_to_bootstrap: 1 during full Galera crash recovery
  • log the local wsrep-recover seqno as evidence only
  • fail closed when no peers are alive and the latest node cannot be proven locally
  • add focused shellspec coverage for unsafe and already-safe pod-0 bootstrap paths

Closes #2996.

Validation

  • shellspec --load-path ./shellspec --shell bash addons/mariadb/scripts-ut-spec/galera_start_spec.sh -> 7 examples, 0 failures
  • bash -n addons/mariadb/scripts/galera-start.sh addons/mariadb/scripts-ut-spec/galera_start_spec.sh
  • git diff --check -- addons/mariadb/scripts/galera-start.sh addons/mariadb/scripts-ut-spec/galera_start_spec.sh

Evidence boundary

This PR fixes the source-level unsafe bootstrap path and has focused unit coverage. It does not claim that the full Galera crash-recovery scenario is runtime-passed until the patched chart/image is validated in a live cluster.

@weicao weicao requested review from a team and leon-ape as code owners July 3, 2026 05:32
@codecov-commenter

codecov-commenter commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (931738c) to head (4acd3fa).

Files with missing lines Patch % Lines
...ddons/mariadb/scripts-ut-spec/galera_start_spec.sh 0.00% 49 Missing ⚠️
...ts-ut-spec/seed_replication_mode_overrides_spec.sh 0.00% 3 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##            main   #2997   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files        140     140           
  Lines      22548   22597   +49     
=====================================
- Misses     22548   22597   +49     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the new pod-0 fail-closed path, the existing non-pod-0 guard still ignores safe_to_bootstrap: 1. In a clean full shutdown where Galera marks a non-pod-0 as the safe node, pod-0 exits with the defer reason and that safe non-pod-0 also returns false to join, so no pod bootstraps the cluster.

@weicao weicao force-pushed the helen/mariadb-galera-bootstrap-fail-closed branch from 601b0ad to cdb834a Compare July 3, 2026 06:39
@weicao

weicao commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in cdb834aec9d68b981a210b4a902356540ce14c87.

Change made:

  • a node with local safe_to_bootstrap: 1 can bootstrap after the no-Primary-peer check, regardless of pod index
  • safe_to_bootstrap: 0 still stays fail-closed on pod-0 and no longer rewrites grastate.dat
  • added shellspec coverage for the clean full-shutdown case where Galera marks a non-pod-0 as safe

Local validation on the rebased commit:

  • shellspec --load-path ./shellspec --shell bash addons/mariadb/scripts-ut-spec/galera_start_spec.sh -> 8 examples, 0 failures
  • bash -n addons/mariadb/scripts/galera-start.sh addons/mariadb/scripts-ut-spec/galera_start_spec.sh
  • git diff --check -- origin/main..HEAD

@weicao weicao added the nopick Not auto cherry-pick when PR merged label Jul 3, 2026
@weicao weicao force-pushed the helen/mariadb-galera-bootstrap-fail-closed branch from cdb834a to 3a4ae56 Compare July 3, 2026 09:54
@weicao

weicao commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed fresh Galera bootstrap regression in 3a4ae56a85ed10ddea4fe9dae13891b11c6cc224.

Change made:

  • treat grastate.dat with seqno=-1 as fresh bootstrap state, not a clean-shutdown election result
  • keep fresh bootstrap single-owner on pod-0 to prevent pod-0/1/2 from each forming their own Primary component
  • preserve the previous clean full-shutdown behavior: a non-pod-0 node with safe_to_bootstrap: 1 and a real seqno can still bootstrap after the no-Primary-peer check
  • keep pod-0 with safe_to_bootstrap: 0 fail-closed and non-mutating

Local validation:

  • shellspec --load-path ./shellspec --shell bash addons/mariadb/scripts-ut-spec/galera_start_spec.sh -> 10 examples, 0 failures
  • bash -n addons/mariadb/scripts/galera-start.sh addons/mariadb/scripts-ut-spec/galera_start_spec.sh
  • git diff --check -- origin/main..HEAD

@weicao weicao force-pushed the helen/mariadb-galera-bootstrap-fail-closed branch from 3a4ae56 to 4acd3fa Compare July 3, 2026 10:08

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nopick Not auto cherry-pick when PR merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(mariadb): fail closed on unsafe Galera crash recovery bootstrap

3 participants