feat(tensilelite): Add TDMLoadWaveSync - #9748
Open
chengchingwen wants to merge 26 commits into
Open
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
cycheng
approved these changes
Jul 23, 2026
cycheng
left a comment
Contributor
There was a problem hiding this comment.
Have had a discussion with Peter offline, refactoring (Analysis+Transform or DataFlow) will be implemented in the follow up PR.
chengchingwen
force-pushed
the
users/chengchingwen/tdmlws
branch
4 times, most recently
from
July 24, 2026 04:55
fe6dfa0 to
6d7a87b
Compare
chengchingwen
force-pushed
the
users/chengchingwen/tdmlws
branch
from
July 27, 2026 02:17
6d7a87b to
effac44
Compare
aazz44ss
self-requested a review
August 5, 2026 02:30
aazz44ss
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID: AIHPBLAS-4146
Motivation
Insert a workgroup barrier between two groups of
tensor_load_to_lds— an urgent, waited-for group and a deferrable prefetch-ahead group. This PR automates that barrier insertion in StinkyTofu.Technical Details
Adds
TDMLoadWaveSyncPass(gfx1250), gated by a new off-by-default TensileLite solution paramTDMLoadWaveSync. It runs after tensorcnt insertion and a kernel-scope CFGBuilder, and inserts a barrier (s_barrier_signal -1/s_barrier_wait -1) between the two TDM groups.s_wait_tensorcntis a trigger; its drained token set S (read fromMemTokenData, now attached to emitted waits byStinkyWaitCntInsertionPass) splits preceding loads into urgent (token ∈ S) vs deferrable (token ∉ S). Wait-group identity == memtoken.ScanState = {frontier of open deferrable groups, workingS}.workingSis the still-urgent subset of S and shrinks by a wait's drained tokens when the scan crosses it (an already-drained token is no longer urgent on that path); the flow ends whenworkingSempties or an urgent load is reached. State is unioned at CFG joins and iterated to a fixpoint — O(blocks × waits × lattice-height), no path enumeration. This covers prologue→loop, steady-state in-loop, diamond, and nested diamond-in-loop control flow.TDM wait-group barriercomment with the wait's drained set) so they are distinguishable from freshly-inserted and plain barriers.StinkyWaitCntInsertionPasstags each emitteds_wait_tensorcntwith the exact drained-token union (WaitDataflow::drainedTensorTokens).TDMLoadWaveSyncis added to the kernel-name parameter set, so two solutions differing only by it do not dedup to one code object.DumpMemTokenIRStructurePass(registered instinkytofu-optas--DumpMemTokenIRStructurePass) dumps a reduced IR view — branch structure + memtoken-candidate instructions + comments — for inspecting wait-group flow.TensileLite wiring:
ValidParameters/GlobalParameters(default[False]),Solution.py(reject unlessScheduleIterAlg=4; disable when TDM is off orNumWaves<=1),KernelWriter.py(thread into the StinkyTofu module options), and a benchmark section intdm_split_gfx1250.yaml.Test Plan
tdm_load_wave_sync.stir: 22 FileCheck cases covering flat/idempotent, cross-BB prologue→loop, steady-state in-loop, diamonds, nested diamond-in-loop, per-arm token-exact barriers, multi-token drained sets with partial-drain arms, and the no-memtoken / intervening-drain negatives.tdm_load_wave_sync_single_wave.stir: NumWaves==1 no-op (pass inserts nothing; matches theSolution.pyguard that disablesTDMLoadWaveSyncwhenNumWaves<=1).waitcnt_insertion_tensor_anchor_ds_read_num_waves_1.stir.tdm_split_gfx1250.yamlexercisesTDMLoadWaveSync: [False, True](MXF8SS Block32, ScheduleIterAlg=4, TDMSplit, NumWaves>1) via the tox kernel-gen tests.
Test Result
Tests passed
Submission Checklist