Skip to content

Size the KV cache to the request rather than model.config.max_seq_len - #1327

Open
risjain wants to merge 1 commit into
fishaudio:mainfrom
risjain:perf/size-kv-cache
Open

Size the KV cache to the request rather than model.config.max_seq_len#1327
risjain wants to merge 1 commit into
fishaudio:mainfrom
risjain:perf/size-kv-cache

Conversation

@risjain

@risjain risjain commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Every decode step attends over the whole allocated cache, so defaulting
max_seq_len to the model maximum (32768) makes a few-hundred-token utterance pay
for positions it never uses.

Isolated decode on a gfx1151 APU (s2-pro bf16):

max_seq_len tok/s
32768 0.84
1024 4.23

End-to-end through the API server with a cloned reference voice: RTF 4.10 ->
1.96
, i.e. a 10-hour audiobook render in 20 h instead of 42 h.

This derives the default from T + max_new_tokens + 64 and adds a MAX_SEQ_LEN
environment variable as an override.

Two things worth reviewing:

  • launch_thread_safe_queue() builds its cache before generate() runs and
    cannot know request lengths, so it keeps the model maximum unless MAX_SEQ_LEN
    is set. Without touching that second site the override does nothing on the queue
    path, because generate() skips its own sizing once _cache_setup_done is set.
  • The size must cover prompt and generated tokens. 1024 passes a quick test
    and then overflows on a long chunk (1167 prompt tokens + up to 1023 generated),
    hence the + 64 margin on top of T + max_new_tokens.

Not AMD-specific, though integrated GPUs feel it worst: they share DRAM bandwidth
with the CPU, and bandwidth is exactly what this wastes. Measured on a Radeon
8060S (gfx1151, 128 GB shared LPDDR5 ~256 GB/s), ROCm 7.13.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Every decode step attends over the whole allocated cache, so defaulting to the
model maximum (32768) makes a few-hundred-token utterance pay for positions it
never uses.

Deriving the default from T + max_new_tokens measured 0.84 -> 4.23 decode tok/s on
a gfx1151 APU, and RTF 4.10 -> 1.96 end-to-end through the API server. A new
MAX_SEQ_LEN environment variable overrides it.

launch_thread_safe_queue() builds its cache before generate() runs and cannot know
request lengths, so it keeps the model maximum unless MAX_SEQ_LEN is set. Without
that second site the override has no effect on the queue path, since generate()
skips sizing once _cache_setup_done is set.

Integrated GPUs see this worst, sharing DRAM bandwidth with the CPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant