Motivation
The load generator derives its keys from a sequential pseudo-random stream: each worker pulls key bytes from a seeded PRNG in generation order. A key is therefore defined by its position in the stream, and the only way to obtain the key at position N is to consume the stream up to N. There is no way to regenerate a specific previously-generated key — or a past transaction — directly.
This blocks any workload that needs to reference keys created earlier in the run — most importantly, reading or querying keys the loadgen already committed (see #671). Today such scenarios require either holding large key sets in memory, or a two-phase procedure (fill in one run, then replay in a second run with the same seed) with no control over how far back a referenced key was created. It also makes the generated key bytes sensitive to the exact order in which the generator consumes its seed, which is fragile as the generator evolves.
What we want
Make each generated key a pure function of its address — (worker, transaction-index, slot) — rather than its position in a stream, so that:
- Any transaction is regenerable by its index — its full content (layout / set sizes, keys, values, and id) is recomputable directly from the index, without replaying the transactions before it.
- Any key is regenerable by its address in O(1) — no replay, no in-memory key store, no second run.
- A workload can reference previously-created keys at a configurable lag within a single run, by regenerating a past transaction t = j − L by its index and recalling its keys.
- The per-transaction layout stays a deterministic function of the transaction index, kept separate from key bytes: the layout decides how many keys a transaction has; the key bytes are a pure function of the address.
- The determinism contract is stronger and simpler: the same seed (with worker count and key size) reproduces the same key for a given address, independent of how the generator consumes its seed.
The ECDSA signature and the envelope timestamp remain non-deterministic and are outside the contract — they are not part of a transaction's identity or content.
Relationship to #671
This is the enabler for #671 (loading the query and committer in parallel): querying previously-created keys during a live commit load means, for a live transaction j, regenerating a past transaction t = j − L by its index and recalling its keys — an O(1) operation with no buffering and no two-phase run. The current sequential stream cannot do this. It also stands on its own as a general determinism/reproducibility improvement for the load generator.
Success criteria
- Keys are a deterministic function of their
(worker, transaction-index, slot) address, reproducible from seed + worker count + key size (the contract is preserved and strengthened, though key bytes are not identical to the current implementation).
- Any transaction (layout, keys, values, id) and any individual key are regenerable by index/address in O(1), without consuming or replaying prior state.
- The per-transaction layout (set sizes) is a deterministic function of the transaction index, decoupled from key identity.
- The existing statistical / distribution properties of the generated workload are preserved.
Motivation
The load generator derives its keys from a sequential pseudo-random stream: each worker pulls key bytes from a seeded PRNG in generation order. A key is therefore defined by its position in the stream, and the only way to obtain the key at position N is to consume the stream up to N. There is no way to regenerate a specific previously-generated key — or a past transaction — directly.
This blocks any workload that needs to reference keys created earlier in the run — most importantly, reading or querying keys the loadgen already committed (see #671). Today such scenarios require either holding large key sets in memory, or a two-phase procedure (fill in one run, then replay in a second run with the same seed) with no control over how far back a referenced key was created. It also makes the generated key bytes sensitive to the exact order in which the generator consumes its seed, which is fragile as the generator evolves.
What we want
Make each generated key a pure function of its address —
(worker, transaction-index, slot)— rather than its position in a stream, so that:The ECDSA signature and the envelope timestamp remain non-deterministic and are outside the contract — they are not part of a transaction's identity or content.
Relationship to #671
This is the enabler for #671 (loading the query and committer in parallel): querying previously-created keys during a live commit load means, for a live transaction j, regenerating a past transaction t = j − L by its index and recalling its keys — an O(1) operation with no buffering and no two-phase run. The current sequential stream cannot do this. It also stands on its own as a general determinism/reproducibility improvement for the load generator.
Success criteria
(worker, transaction-index, slot)address, reproducible from seed + worker count + key size (the contract is preserved and strengthened, though key bytes are not identical to the current implementation).