Problem
A restore ActionSet with an All pod target creates one postReady Job per selected pod and starts all Jobs concurrently. Some distributed engines require a snapshot to be restored on every node but require those node-level recovery operations to run sequentially.
In a frozen 3-node Qdrant diagnostic run, all three snapshot upload requests returned HTTP 200 within a 348 ms window. One peer then stopped consensus while shard recovery and transfer state overlapped. Data readback remained complete (40/40), but the cluster stayed Updating.
Required contract
- preserve parallel execution as the default
- allow an ActionSet to opt into serial postReady target execution
- start only the first target Job
- start Job N+1 only after Job N completed successfully
- stop immediately when the active Job fails
- remain retry-safe across controller reconciles and partial Job creation
Non-goals
- fixed sleeps in engine scripts
- changing prepareData PVC restore policy
- forcing all engines to serialize
Problem
A restore ActionSet with an
Allpod target creates one postReady Job per selected pod and starts all Jobs concurrently. Some distributed engines require a snapshot to be restored on every node but require those node-level recovery operations to run sequentially.In a frozen 3-node Qdrant diagnostic run, all three snapshot upload requests returned HTTP 200 within a 348 ms window. One peer then stopped consensus while shard recovery and transfer state overlapped. Data readback remained complete (40/40), but the cluster stayed Updating.
Required contract
Non-goals