Skip to content

feat(tensilelite): Add TDMLoadWaveSync - #9748

Open
chengchingwen wants to merge 26 commits into
ROCm:developfrom
chengchingwen:users/chengchingwen/tdmlws
Open

feat(tensilelite): Add TDMLoadWaveSync#9748
chengchingwen wants to merge 26 commits into
ROCm:developfrom
chengchingwen:users/chengchingwen/tdmlws

Conversation

@chengchingwen

@chengchingwen chengchingwen commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

JIRA ID: AIHPBLAS-4146

Motivation

Insert a workgroup barrier between two groups of tensor_load_to_lds — an urgent, waited-for group and a deferrable prefetch-ahead group. This PR automates that barrier insertion in StinkyTofu.

Technical Details

Adds TDMLoadWaveSyncPass (gfx1250), gated by a new off-by-default TensileLite solution param TDMLoadWaveSync. It runs after tensorcnt insertion and a kernel-scope CFGBuilder, and inserts a barrier (s_barrier_signal -1 / s_barrier_wait -1) between the two TDM groups.

  • Trigger / classification. Each s_wait_tensorcnt is a trigger; its drained token set S (read from MemTokenData, now attached to emitted waits by StinkyWaitCntInsertionPass) splits preceding loads into urgent (token ∈ S) vs deferrable (token ∉ S). Wait-group identity == memtoken.
  • Backward monotone dataflow. A per-trigger worklist scans backward, carrying a ScanState = {frontier of open deferrable groups, workingS}. workingS is the still-urgent subset of S and shrinks by a wait's drained tokens when the scan crosses it (an already-drained token is no longer urgent on that path); the flow ends when workingS empties or an urgent load is reached. State is unioned at CFG joins and iterated to a fixpoint — O(blocks × waits × lattice-height), no path enumeration. This covers prologue→loop, steady-state in-loop, diamond, and nested diamond-in-loop control flow.
  • No-redundant / tagging. A pre-existing barrier already at a split suppresses insertion; such barriers are annotated in the emitted asm (a TDM wait-group barrier comment with the wait's drained set) so they are distinguishable from freshly-inserted and plain barriers.
  • Supporting change. StinkyWaitCntInsertionPass tags each emitted s_wait_tensorcnt with the exact drained-token union (WaitDataflow::drainedTensorTokens).
  • Naming. TDMLoadWaveSync is added to the kernel-name parameter set, so two solutions differing only by it do not dedup to one code object.
  • Debug aid. DumpMemTokenIRStructurePass (registered in stinkytofu-opt as --DumpMemTokenIRStructurePass) dumps a reduced IR view — branch structure + memtoken-candidate instructions + comments — for inspecting wait-group flow.

TensileLite wiring: ValidParameters/GlobalParameters (default [False]), Solution.py (reject unless ScheduleIterAlg=4; disable when TDM is off or NumWaves<=1), KernelWriter.py (thread into the StinkyTofu module options), and a benchmark section in tdm_split_gfx1250.yaml.

Test Plan

  • tdm_load_wave_sync.stir: 22 FileCheck cases covering flat/idempotent, cross-BB prologue→loop, steady-state in-loop, diamonds, nested diamond-in-loop, per-arm token-exact barriers, multi-token drained sets with partial-drain arms, and the no-memtoken / intervening-drain negatives.
  • tdm_load_wave_sync_single_wave.stir: NumWaves==1 no-op (pass inserts nothing; matches the Solution.py guard that disables TDMLoadWaveSync when NumWaves<=1).
  • Producer-side memtoken checks in waitcnt_insertion_tensor_anchor_ds_read_num_waves_1.stir.
  • TensileLite: tdm_split_gfx1250.yaml exercises TDMLoadWaveSync: [False, True]
    (MXF8SS Block32, ScheduleIterAlg=4, TDMSplit, NumWaves>1) via the tox kernel-gen tests.

Test Result

Tests passed

Submission Checklist

@therock-pr-bot

therock-pr-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass
Forbidden Files ✅ Pass
🧪 Unit Test ✅ Pass
🔎 pre-commit ✅ Pass
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled
🤖 therock-pr-bot ✅ Pass

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@cycheng cycheng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have had a discussion with Peter offline, refactoring (Analysis+Transform or DataFlow) will be implemented in the follow up PR.

@chengchingwen
chengchingwen force-pushed the users/chengchingwen/tdmlws branch 4 times, most recently from fe6dfa0 to 6d7a87b Compare July 24, 2026 04:55
@chengchingwen
chengchingwen force-pushed the users/chengchingwen/tdmlws branch from 6d7a87b to effac44 Compare July 27, 2026 02:17
@aazz44ss
aazz44ss self-requested a review August 5, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants