In-memory metastore overlay over MySQL (Phase 1, #347)

[em3s]

Summary

Phase 1 of metastore consolidation (#347): extend LocalBackedJdbcHashLabel with an operational overlay layer so the metastore is served from a fast store while MySQL becomes a read-only fallback.

This PR scopes the overlay to an in-memory ByteArrayIndexedLabel — a deploy/rollback-safe stepping stone that needs no HBase provisioning. Durable state stays in MySQL (mirrored on writes). Swapping the overlay for a persistent HBase-backed label is a later step.

Part of #347

Changes

• LocalBackedJdbcHashLabel — three layers: local seed (in-memory), operational overlay (in-memory), MySQL base
  • Write: overlay is the target; all ops mirror to MySQL for rollback safety (useJdbcMetastore=true)
  • Read/scan: overlay wins on (src,tgt) dedup; MySQL rows appear only when the overlay has no entry
  • Scan: indexName=null → __default__ prefix-scan index for both indexed backends; MySQL keeps the original filter
  • Count: local + overlay only — MySQL cannot count (JdbcHashLabel returns a -1 sentinel)
  • useJdbcMetastore=false bypasses MySQL entirely (overlay-only)
• GraphDefaults / Graph — add consolidatedStore: ByteArrayStore (dedicated in-memory overlay store, separate from the local seed store)
• GraphConfig — add useJdbcMetastore toggle (default true)
• LocalBackedJdbcHashLabelOverlayTest — mockk unit tests covering write routing, DELETE mirroring, read dedup, count (overlay + local, MySQL never consulted), and lock delegation

Merge policy

Op            | local (seed) | overlay (memory) | MySQL (base) | Rule
--------------|--------------|------------------|--------------|------------------------------
INSERT/UPDATE | useLocal=T   | useLocal=F       | mirrored     | overlay is write target
DELETE        | useLocal=T   | useLocal=F       | mirrored     | both layers deleted → no resurrection
read/scan     | always       | always           | fallback*    | overlay wins on (src,tgt) dedup
count         | always       | always           | never        | local + overlay; MySQL cannot count

* MySQL merge is skipped when useJdbcMetastore=false.

How to Test

./gradlew :engine:test — 621 tests pass. LocalBackedJdbcHashLabelOverlayTest covers write routing, DELETE mirroring, read dedup, count (MySQL never consulted), and lock delegation; LocalBackedJdbcHashLabelTest covers __default__ scan routing and the no-sentinel count invariant.

AI Assistance

• This PR was written with AI assistance.
  • Tool / model: Claude Code (Opus 4.8)

[em3s]

Phase 2 Migration Plan

Phase 1 overlay is now live. To migrate MySQL → HBase and remove MySQL:

Step 1 — Seed HBase from MySQL (insert-only)

For each metadata label (service, storage, label, query, alias):

for each database (tenant):
    label.getAll(database)          // reads from MySQL (fallback layer)
    → for each entity:
        consolidatedLabel.mutate(INSERT, failOnExist=false)   // setnx — skip if HBase already has it

After this, every MySQL row exists in HBase. Phase 1 HBase writes are never overwritten because failOnExist=false (= setnx) preserves the newer HBase version.

Step 2 — Remove MySQL from the read path

Once the seed is verified (reconcile MySQL key count vs HBase key count):

• merge(hbase, mysql, keyOf) → return hbase directly (drop globalLabel reads)
• Remove globalLabel field from LocalBackedJdbcHashLabel
• Remove DELETE mirroring to MySQL

Step 3 — Remove JDBC/H2 entirely (Phase 3)

• Drop LocalBackedJdbcHashLabel, JdbcHashLabel, LocalStorage, JdbcStorage
• Route local_backed_metastore labels directly to HBaseIndexedLabel
• Remove MySQL/H2/Exposed/HikariCP dependencies

[em3s]

Reverse-merged main (7d731fd). Conflicts in GraphDefaults.kt and LocalBackedJdbcHashLabel.kt resolved: local layer is now ByteArrayIndexedLabel (#413).

count() now merges local + HBase overlay only — MySQL can't count (returns a -1 sentinel), so it's excluded. Both count tests updated to match. :engine:test green.

[em3s]

Updated: the overlay is now an in-memory ByteArrayIndexedLabel, not HBase-backed. No HBase table provisioning is required, and the change is deploy/rollback-safe.

As-is (default use-jdbc-metastore: true) is not breaking — writes go to the in-memory overlay + mirror to MySQL, reads merge both (overlay wins), so nothing becomes invisible. Durable state stays in MySQL.

Deploy → rollback is safe — MySQL holds every operational write, and the overlay is volatile with no persistent footprint, so rolling back to the previous version loses nothing.

use-jdbc-metastore: false is breaking — it drops the MySQL fallback, so MySQL-only metadata disappears; the system must be migrated first.

Next steps (later PRs):

• Migrate MySQL → HBase
• Swap the in-memory overlay for a persistent HBase-backed label
• Switch to false (HBase-only)

[em3s]

Optimistic merge.