[WIP] feat(deepseek): land V4 L1 KV cache on heterogeneous flat kv pools - #750
[WIP] feat(deepseek): land V4 L1 KV cache on heterogeneous flat kv pools#750SimonCqk wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a11e4ada3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
98e28a5 to
5a35355
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a35355ff0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
442495c to
b777c65
Compare
b777c65 to
b85fed1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b85fed17e4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b85fed1 to
f603a07
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f603a07a69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| lcm_spec=( | ||
| lcm_setup.draft if use_lcm_v4 and lcm_setup is not None else None | ||
| ), |
There was a problem hiding this comment.
When DeepSeek V4 speculative/MTP decoding enables this separate draft LCM pool and the scheduler recycles a parent page, ModelExecutor.zero_flat_cache_pages() sanitizes only self.token_to_kv_pool; it never calls the sanitizer on self.draft_token_to_kv_pool, even though the new draft pool also sets flat_kv_requires_page_zeroing=True and uses the same scheduler page IDs in an independent backing arena. Reused draft SWA/compressor pages can therefore retain bytes from the previous owner and contaminate draft computation where a step does not overwrite the entire aliased page. Sanitize the corresponding draft-group pages as well and record the completion event after both pools are cleared.
Useful? React with 👍 / 👎.
e1bc75e to
4be215b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4be215b7c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) | ||
|
|
||
|
|
||
| def _prepare_deepseek_v4( |
There was a problem hiding this comment.
This introduces a new heterogeneous FlatKV/LCM architecture with important operational constraints, but no documentation was added or updated; a repo-wide search also confirms that the migration guide cited by the commit, docs/serving/deepseek-v4-flat-kv-cache-migration.md, does not exist. Add that guide or update an existing serving document so operators can discover the required Flat scheduler build and unsupported L2/PD combinations.
AGENTS.md reference: AGENTS.md:L13-L13
Useful? React with 👍 / 👎.
Signed-off-by: SimonCqk <cqk0100@gmail.com>
4be215b to
2eddc81
Compare
|
@SimonCqk let's wait for #864 @wangbo981016 |
2 participants
Summary
Migrate DeepSeek V4 L1/device KV cache management to heterogeneous flat block pools while preserving the radix path as a compatibility and differential oracle.
The new design makes the cache plan the single authority for group geometry, physical pools, target/draft ownership, table layout, and device metadata. Model-agnostic runtime components consume only generic cache capabilities and table/lifecycle interfaces.
Key Changes
BlockPoolSetandKvCacheCoordinator.Scope
This PR covers device-side/L1 KV cache management only. It does not add L2/CPU offload, kvstore integration, group-aware PD transfer, or global radix removal.