Paper: dagster-slurm: Dagster Assets on Slurm-Managed Scientific Workflows

[HPicatto]

Submission of the paper dagster-slurm: Dagster Assets on Slurm-Managed Scientific Workflows.

Paper directory: papers/dagster_slurm/.

dagster-slurm extends Dagster's control plane to Slurm-managed HPC clusters, so the same asset source code runs on a laptop, in CI, against a containerized Slurm cluster, and on a production HPC cluster over SSH. It is evaluated through a production firm-discovery pipeline that has run more than 9,600 Slurm jobs on the TU Wien DataLAB cluster.

• Builds cleanly with the mystmd CLI (HTML and the configured Typst PDF export).
• Corresponding author: Hernan Picatto.
• All content is confined to the paper directory.

Happy to add reviewers or adjust as needed.

[github-actions]

Curvenote Preview

Directory	Preview	Checks	Updated (UTC)
papers/dagster_slurm	🔍 Inspect	❌ 60/61 checks passed (15 optional)	Jun 21, 2026, 3:54 AM

[chongshenng]

Hello @HPicatto, I am Chong Shen Ng (@chongshenng). I will serve as the editor for your submission. Please reach out to me if any assistance needed 😊.

[jasdeepbhalla]

I can review this paper. Please add me as a reviewer. Thanks

[jasdeepbhalla]

Thanks for this. I have left inline suggestions on the specific lines.

Three things stand out:

• build-preview / check is failing on a single item. Looks like Citation does not have DOI: airflow.
• The evaluation never says what the abstraction costs: median added latency per job from packaging, transfer, and Pipes polling, against a bare sbatch, would materially strengthen the paper.
• The abstract reports portability checks on VSC-5 and Leonardo that line 329 says are no longer evidenced, so that claim needs rewording or one confirming job.

Nice work on reading operational totals from sreport rather than Dagster's own run log, and on the reproducibility boundary paragraph.

[jasdeepbhalla]

The evaluation covers tested, operated at scale, and observable, but not what the abstraction costs.

• What is the median added latency per job from environment packaging, payload transfer, and Pipes polling, compared to a bare sbatch of the same payload?
• And how much of that does the pre-deployed-environment path remove?

You already have the instrumentation that produced 9,626 jobs of accounting data, and there are 2,500 words of headroom.

Without this number the paper describes the system rather than evaluates it.

[jasdeepbhalla]

Suggested change

	The adapters are trained in a chain, each starting from the previous one, with a fraction of earlier-region data replayed; on held-out test domains the adapter is intended to retain coverage of earlier language groups while adding the new one, and our internal evaluation reports eval loss between 0.047 and 0.055 down the chain.
	The adapters are trained in a chain, each starting from the previous one, with a fraction of earlier-region data replayed.
	On held-out test domains, extraction quality on earlier language groups changes by FILL IN (metric, before and after the next link) while the new group reaches FILL IN, indicating that the chain adds coverage without measurable regression.

If you do not have that measurement, delete the claim entirely rather than reporting the raw eval loss. A loss value with no task, scale, or baseline is not interpretable, and "is intended to retain coverage" is not a claim.

[jasdeepbhalla]

Suggested change

	Over the ten-week accounting window in @tbl:ops it submitted 9,626 Slurm jobs, of which 79.5% completed and 9.9% failed for reasons other than memory or time limits (with 64 out-of-memory and 57 timeout terminations counted separately; a further 9.3% were cancelled, which on this cluster includes deliberate reruns for stack consistency and retry supersessions rather than only errors), and it consumed 2,509 GPU-hours and 118,914 CPU core-hours.
	Over the ten-week accounting window in @tbl:ops it submitted 9,626 Slurm jobs and consumed 2,509 GPU-hours and 118,914 CPU core-hours.
	Of those jobs, 79.5% completed, 9.9% failed for reasons other than memory or time limits, 9.3% were cancelled, 0.7% hit out-of-memory, and 0.6% hit the walltime limit.
	The cancelled jobs are largely deliberate reruns for stack consistency and retry supersessions rather than errors.
	The largest failure category resolves to FILL IN (top three causes, e.g. transient S3 fetch errors, model-server startup timeouts, node-local disk exhaustion), which the scheduler metadata surfaced in Dagster without additional instrumentation.

That last sentence turns the paper's weakest number into its best demonstration.

[jasdeepbhalla]

Suggested change

	Even after deduplicating to one page per domain and page type, running extraction over the resulting candidate pages without first applying the registry country filter would exceed 1,100 A100-hours, with most of it spent on out-of-scope domains.
	Even after deduplicating to one page per domain and page type, roughly FILL IN million candidate pages remain; extracting over all of them at the 15,000 to 22,000 pages per hour of @tbl:tiers, without first applying the registry country filter, would exceed 1,100 A100-hours, with most of it spent on out-of-scope domains.

This is the strongest argument for the multi-tier design and currently the one number a reader cannot check.

[jasdeepbhalla]

Suggested change

	**Observability.** Slurm job identifiers, CPU efficiency, memory use, and node-hours surface as Dagster metadata where Slurm accounting is available, and Ray clusters stream their output back through Pipes.
	Conventional Dagster asset checks and alerting therefore operate over those metadata and failure signals without modification, a property that separate script-based workflows often lose or must rebuild manually.
	**Observability.** Slurm job identifiers, CPU efficiency, memory use, and node-hours surface as Dagster metadata where Slurm accounting is available, and Ray clusters stream their output back through Pipes.
	Across the accounting window, FILL IN% of submitted jobs returned a complete scheduler metadata record; the remainder ran and streamed logs but lacked `sacct` fields.
	Conventional Dagster asset checks and alerting therefore operate over those metadata and failure signals without modification, a property that separate script-based workflows often lose or must rebuild manually.

Tested has CI timings and operated-at-scale has sreport totals.

Observability is the one dimension of the three with nothing measured behind it.

[jasdeepbhalla]

Suggested change

	We report this as production operational experience rather than as a scheduler-scalability benchmark, which would require a controlled study of submission throughput and concurrent-job limits that is outside this paper's scope.
	We report this as production operational experience rather than as a scheduler-scalability benchmark, which would require a controlled study of submission throughput and concurrent-job limits that is outside this paper's scope.
	**Limitations.** Five constraints bound these results.
	Data movement is left to the user, so cross-target portability of asset code depends on a deployment-mode-aware path convention rather than on anything the package enforces.
	Scheduler accounting metadata requires `sacct` to be exposed at the site; where it is not, jobs run and logs stream but those fields are absent.
	Clean restart depends on asset-level idempotency, and assets without an idempotent output convention recompute from the start.
	Production measurements come from one cluster and one ten-week window, not from a multi-site or lifetime study.
	The Spark launcher, session-based allocation reuse, and heterogeneous jobs are experimental and are not covered by any measurement reported here.

All five facts are already in the paper at lines 83, 101, 100, 329, and 166.

Collecting them makes the paper's honesty a visible strength rather than something the reader has to assemble.

[jasdeepbhalla]

Suggested change

	The integration is organized in three layers, shown below, and mapped onto execution targets in @fig:arch.
	The integration is organized in three layers (@fig:layers) and mapped onto execution targets in @fig:arch.

[jasdeepbhalla]

Suggested change

	:::{figure} software_layers.png
	The three software layers.
	:::{figure} software_layers.png
	:label: fig:layers
	The three software layers.

[jasdeepbhalla]

Suggested change

	The diagram below shows the stages.
	@fig:pipeline shows the stages.

[jasdeepbhalla]

Suggested change

	:::{figure} pipeline.png
	Stages of one (region, crawl) run.
	:::{figure} pipeline.png
	:label: fig:pipeline
	Stages of one (region, crawl) run.