Separate Bulletin user-data storage from the main database — drop NVMe requirement, scale on SSDs

[mudigal]

Requirement (H2)

Separate Bulletin Chain user-data storage from the main (state) database so that BC storage no longer requires NVMe and can be scaled on standard SSDs.

Priority: plan for this irrespective of post-launch scale — it is not contingent on demand growth. Per the H2 director sync, Gav confirmed SSD support for BC is needed this year, ahead of Web3 Storage, priority wise.

Background

BC user data is persisted via Substrate transaction indexing (sp_io::transaction_index::index / renew, see pallets/transaction-storage/src/lib.rs). This bulk, cold, throughput-oriented data currently shares the node database with the small, latency-sensitive state trie that genuinely benefits from NVMe. The goal is to split these tiers so the bulk user data can live on cheaper SSDs.

Why it matters — metanode + scale economics

1. Metanode. A single operator (potentially a single node) will run RC validation plus collation of all 5 system chains — one of which is BC, with its full state and user data. That is 5× chains to sync/operate, sharply increasing NVMe demand per operator.
2. Fleet scale. With metanode, every validator (~300–500 nodes) runs BC. The NVMe→SSD cost delta then scales across 300–500 nodes instead of 5–8 collators — this is where the real saving is. (Side benefit: 300–500 replicas give heavily over-provisioned data availability, which simplifies durability.)
3. Storage growth. As BC scales (2s, potentially 500ms blocks), per-node storage grows atleast 3x from current ~8 TB. The NVMe-vs-SSD cost gap widens with increase in capacity.
4. Cost. ~$250/node was previously raised for additional NVMe; SSD roughly halves per-node storage cost — but the aggregate across the metanode fleet, at scaling capacity, is the material number.

Primary deliverable — cost/benefit analysis

A documented analysis leadership can decide on. It should show whether the separation is worth doing, covering:

• NVMe vs standard SSD $/node and fleet total at 8 / 24 / 100 TB, across 5–8 collators vs 300–500 metanode operators.
• The engineering cost of the storage separation itself.
• Performance implications — does BC's storage-proof generation, Bitswap serving, and sync tolerate SSD IOPS/latency at 24–100 TB? What (if anything) must stay on NVMe?

Input data (from the director sync): 300–500 nodes/operators each run RC validation (sync RC) + collation of 5 system chains (sync all 5), one being BC with full state + user data; yielding 300–500 DA replicas (over-provisioned).

Scope of work

• Architect separation of the indexed transaction-storage data (user-data blobs) from the main state DB, allowing the bulk tier on a distinct/cheaper volume.
• Evaluate node DB backend options (ParityDB / RocksDB) for placing transaction data on a separate disk/tier; determine whether a dedicated blob store is warranted.
• Classify workloads: which require NVMe (state trie, consensus/import hot path) vs. which tolerate SSD (bulk cold user data, Bitswap serve, proof-generation reads).
• Provide a configuration to run BC storage on standard SSDs and benchmark vs NVMe (throughput, proof-check latency, sync time, Bitswap serve) at representative capacities.
• Deliver the cost/benefit analysis and a go / no-go recommendation.

Acceptance criteria

• BC can run its bulk user data on standard SSDs with NVMe not required for that tier (or a data-backed recommendation not to, if the analysis shows so).
• Cost/benefit analysis delivered (per-node + fleet at 300–500 nodes, at 8/24/100 TB).
• Benchmarks demonstrate SSD is acceptable for BC's read/serve/proof workloads, or identify precisely what must remain on NVMe.

Open questions

• Does storage-proof generation / Bitswap serving tolerate SSD latency/IOPS at 24–100 TB?
• Backend support for split volumes (ParityDB vs RocksDB); dedicated blob store needed?
• Metanode interaction: shared disk budget across RC + 5 system chains on one operator.
• Retention/pruning implications at 100 TB.

Related

• Metanode initiative (validator + collation + storage bundle).
• Bulletin scaling / clusters (Elastic scaling & multi-core ("clusters") for Bulletin #623), 2s / 500ms blocks.
• Ties to Bulletin mainnet post-launch scaling (H2 Goal Bump docker/build-push-action from 4 to 6 #2).

[x3c41a]

Design doc is ready https://docs.google.com/document/d/11SdmcSH-c0h7h4lVJMtIp6pT2mJHHVuHbZBbnKYEqww/edit?tab=t.0
It contains the options, the costs, and what has to stay on NVMe.

Some key points:

• The saving only matters at fleet scale (300-500 metanode nodes), not a few collators; savings grow as storage grows.
• SSD is ~33% cheaper than NVMe ($120 vs $180/TB). HDD is ~15x cheaper which is the real lever
  for cold data, once the benchmark confirms HDD is fast enough for reads and proofs. (TBC),
• Drive wear is not a concern: at ~110 GiB/day the chain writes 15-35x less than a cheap SSD
  is built to handle.

Prototype PRs:

• Add optional per-column storage path parity-db#251 - per-column storage path, the actual split mechanism (tested).
  It's a node/DB change, not a runtime change, so the pallet, proofs, and Bitswap don't change.
• Add storage-tier benchmark harness #647 - benchmark harness to check SSD/HDD holds block
  deadlines vs NVMe.

Next steps:

1. Review the doc and agree on the design. @bkontur @karolk91 @mudigal
2. Then ask SREs to provision a tiered test env (NVMe + SSD, 8 TB, then 24 TB, then 100 TB).
3. Run the benchmark to confirm what can move off NVMe.