Add Cloud Native AI Scheduling Challenges Whitepaper#2164
Conversation
github-actions
Bot
added
needs-triage
andreyvelich
left a comment
andreyvelich
left a comment
There was a problem hiding this comment.
Thanks for this effort @rajaskakodkar!
Overall, looks great, I left few thoughts.
processLabelRule previously computed `shouldApply = !foundNamespace`
unconditionally, so a `kind: label` rule with `matchCondition: AND`
behaved identically to a `NOT` rule. Paired NOT/AND rules (e.g. apply
`needs-triage` when no `triage/*` exists, remove it when one does) ended
up firing in exactly the wrong situations: on a fresh PR with no labels
the labeler would add `needs-triage`/`needs-kind`/`needs-group` and
immediately remove them in the same run, and when a `triage/*` label was
later added manually via the UI the paired `needs-*` label would never
be cleared.
Also teach the label-rule `match` parser to understand a single level of
comma-separated brace alternation such as `{toc,tag/*,sub/*}`.
`filepath.Match` on its own does not support braces, so previously such
a pattern only matched a literal label whose name began with `{`.
Adds focused tests for both behaviors, including a regression test that
mirrors the cncf/toc#2164 scenario.
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Riaan Kleinhans <riaankleinhans@gmail.com>
|
|
||
| | Stage | Primary Resources | Duration | Scheduling Characteristics | | ||
| | :---- | :---- | :---- | :---- | | ||
| | Data Preparation | CPU, storage I/O, network | Minutes to hours | Parallelizable, event-driven, no gang requirement | |
There was a problem hiding this comment.
It's helpful to denote that there is no gang requirement -- could you please rephrase your suggesetion @andreyvelich
|
|
||
| ### **Traditional HPC Schedulers: Task-Level Scheduling** | ||
|
|
||
| In traditional high-performance computing (HPC) environments, schedulers such as Slurm operate at the task level (also called a rank or process group member). |
There was a problem hiding this comment.
Do we need link for Slurm? are there other schedulers in that space? Is Slurm the most used one? Asking for context.
There was a problem hiding this comment.
Not a blocker - recommend publishing if all comments are resolved and this can be included in the next version
|
|
||
| This model naturally supports gang scheduling, topology-aware placement, and reservation-based execution. These capabilities are particularly well-suited for model training workloads. | ||
|
|
||
| In HPC-style schedulers, the scheduling unit is a task within a job, with all tasks scheduled together. |
There was a problem hiding this comment.
I would love to see a diagram here, to make the relationship between the concepts such as tasks and jobs more easily understandable for people that haven't use these systems in the past.
There was a problem hiding this comment.
Not a blocker - recommend publishing if all comments are resolved and this can be included in the next version
| * New data arrives in a storage bucket → trigger a data preparation pipeline | ||
| * A model training job completes → trigger an evaluation job | ||
| * An upstream job fails → trigger a notification or retry | ||
|
|
There was a problem hiding this comment.
having a diagram here might also help to solidify how concepts connect to each other.
There was a problem hiding this comment.
Though helpful, not a blocker - recommend publishing if all comments are resolved and this can be included in the next version
There was a problem hiding this comment.
A diagram here would be pretty straight-forward on the text mentioned. I wonder if it would add any value.
|
Folks, congratulations, these papers are great reads. I've added some comments to improve the reader experience, but besides that this looks awesome. I've noticed that some terms like (all-reduce) have no references and for the non ML/AI engineer those thing might require some concrete references, as they are mentioned in several places. I can't wait for this to get published. |
riaankleinhans
added
toc
mesutoezdil
left a comment
mesutoezdil
left a comment
There was a problem hiding this comment.
Two small fixes: a broken markdown link in paper 3 (gang scheduling section) and missing references for the all-reduce term, which salaboy noted has no citations for readers unfamiliar with ML.
|
|
||
| | Stage | Primary Resources | Duration | Scheduling Characteristics | | ||
| | :---- | :---- | :---- | :---- | | ||
| | Data Preparation | CPU, storage I/O, network | Minutes to hours | Parallelizable, event-driven, no gang requirement | |
There was a problem hiding this comment.
Could you clarify please @angellk? As I mentioned data processing jobs might also require gang-scheduling requirements to ensure driver and executors have sufficient capacity to run.
The only differences is that such jobs can tolerate executor failures, which is uncommon for training jobs.
This is what I ment by elastic by nature
|
+1 on @kfaseela point about keda in the scalability row. the whitepaper covers scheduling well but gpu-aware autoscaling is the runtime counterpart once a workload is scheduled, something still needs to scale it based on actual gpu load. kubernetes hpa can't see gpu utilization, so the common path is dcgm-exporter → prometheus → keda. that works but adds latency. i maintain keda-gpu-scaler which reads nvml directly from a daemonset — same approach documented in kedacore/keda#7538. might be worth a mention in the solutions section alongside the prometheus-based patterns. also +1 on @archlitchi re: HAMi. i've contributed to HAMi (#1893) and the topology-awareness piece is relevant to the numa/scheduling challenges covered in section 4. |
4133a2b to
23510aa
Compare
Signed-off-by: Rajas Kakodkar <rajaskakodkar16@gmail.com>
23510aa to
81d3652
Compare
angellk
left a comment
angellk
left a comment
There was a problem hiding this comment.
lgtm! thanks for addressing all the blocker comments @rajaskakodkar - any outstanding comments can be saved for the next version of this whitepaper
11 participants
Adds the Cloud Native Scheduling Challenges Whitepaper