Paper: VOGON POETRY: Revisiting data engineering concepts with an eye for maximizing value in an AI-driven value chain

[shauryashaurya]

AI is fast becoming the core value engine. The differentiator is no longer the model, but the ability to address determinism, correctness, latency and unit costs. Data Engineering must be looked at from this lens.

In this fast evolving ecosystem, the expert-novice data engineer gap cannot be addressed by building tool fluency, but instead building a high quality understanding of a pedagogy of core concepts - a core set of transferrable mental models that unlock analysis and prediction of behaviour across tools and systems - prevalent and emergent.

This paper organizes these mental models as concepts, classified into "instincts" - invariant ideas that govern a class of decisions, grouped into "families" - that form the general heuristic idea in data engineering. Honestly, when read together - these sound like a poem that would make Prostetnic Vogon Jeltz slurp his axlegrurts hagrilly. Hence the name.

The families function as a dependency structure rather than a taxonomy. A decision in one family, such as file layout under "storage as a substrate," propagates predictable consequences in others, such as scan cost under "mechanical sympathy" and shuffle volume under "distribution."

The author claims that familiarity with these concepts first, then attaching tools to them, yields faster transfer to unfamiliar systems than tool-first instruction, because these mental models largely act as invariants and remain stable while interfaces change. This paper presents the map, the concept graph, and the reasoning each instinct supports, with banking and reinsurance as the primary worked domains.

By order of Galactic Hyperspace Planning Council,
Recitation shall now begin, without mercy.
Resistance is useless.

Hi @shauryashaurya , I am Sanhita Joshi (@sanhitamj). I will serve as the Editor for this submission. Let me know if I can help with any questions, queries related to Proceedings. I am also looking into whether we need 2 independent PRs or one is sufficient.

[shauryashaurya]

@sanhitamj has edited this comment, to avoid confusion. This is PR for only one paper.

The first paper is an in person paper presentation:

Paper: VOGON POETRY: Revisiting data engineering concepts with an eye for maximizing value in an AI-driven value chain

@sanhitamj has removed part of this comment about the other paper.

[sanhitamj]

@shauryashaurya will you please make another PR for the Silmaril paper please?

[shauryashaurya]

@shauryashaurya will you please make another PR for the Silmaril paper please?

Have created a separate PR for the Silmaril paper now: #1266

[jameslamb]

👋🏻 hi @shauryashaurya I'm James, one of the reviewers for this paper. I went through tonight and left a very shallow review with a few small comments.

Please especially remove the Silmaril stuff now that there's a new PR for that (you opened the PR but left those files here).

I'll read the paper and review more thoroughly over the next few days. Thanks for the opportunity.

[jameslamb]

Have created a separate PR for the Silmaril paper now: #1266

This and the other Silmaril files should be removed from the PR, please.

[jameslamb]

Suggested change

	A static figure cannot convey the dependency structure that is the central claim of this work; the interactive form lets the reader trace those dependencies directly. The viewer and the underlying data are available online at [Vogon Poetry on GitHub](https://github.com/shauryashaurya/vogon-poetry), and readers are invited to explore the concepts there. The 80 concepts and their typed relationships are published as an interactive concept graph that accompanies this paper. The viewer renders the 8 families, 18 instincts, and their concept nodes as a force-directed graph in the browser, colored by family and clustered by instinct, with the cross-family edges that encode where one invariant constrains another drawn explicitly. Readers can switch between layouts, filter by family or link type, isolate a single instinct and expand its concepts into readable cards, and search for individual nodes. Each node exposes its definition, a worked example on the canonical schema, and its links to related concepts.
	A static figure cannot convey the dependency structure that is the central claim of this work; the interactive form lets the reader trace those dependencies directly.
	The viewer and the underlying data are available online at [Vogon Poetry on GitHub](https://github.com/shauryashaurya/vogon-poetry), and readers are invited to explore the concepts there.
	The 80 concepts and their typed relationships are published as an interactive concept graph that accompanies this paper.
	The viewer renders the 8 families, 18 instincts, and their concept nodes as a force-directed graph in the browser, colored by family and clustered by instinct, with the cross-family edges that encode where one invariant constrains another drawn explicitly.
	Readers can switch between layouts, filter by family or link type, isolate a single instinct and expand its concepts into readable cards, and search for individual nodes.
	Each node exposes its definition, a worked example on the canonical schema, and its links to related concepts.

Would you please consider breaking everything up into at most 1 sentence per line? That should be rendered identically (most markdown renderers group consecutive non-empty lines as a paragraph with appropriate wrapping).

It'd make leaving review comments / suggestions and reading the diff much easier.

[jameslamb]

Please do a pass for adding blank lines between different elements (e.g. between section headings and code blocks).

This reduces the risk of some types of rendering issues and also makes the content a little easier to review here in GitHub.

See https://yihui.org/en/2021/06/markdown-breath/ for more

[jameslamb]

Was there meant to be some code here?

[jameslamb]

Also please do address the failing CI:

No config file present for path "papers/SHAURYA_AGARWAL_Vogon_Poetry"
Create papers/SHAURYA_AGARWAL_Vogon_Poetry/myst.yml and ensure it includes a valid project.id

(build link)

Also I think it's expected that you provide configuration for rendering the paper with MyST: https://github.com/scipy-conference/scipy_proceedings#write-your-paper

[shauryashaurya]

Hiya James All valid comments! Please give me a couple of days to implement everything. Couple of doubts: 1. I am worried that removing the Silmaril docs will delete the whole PR - because GitHub took the whole diff and made it one PR. Is there a way for us to ignore the other documents? Or if you have guidance for me - I'll happily follow up. 2. I thought MyST was a soft requirement - due to paucity of time I just keyed everything in GH Markdown (familiarity, comfort - mea culpa) - let me go through MyST - I'll carefully redraw the doc in MyST at my earliest. I think the ETA would be Monday the 27th, I hope it'll be ok. I greatly appreciate you going through the paper. Thank you tons!

…

On Wed, Jul 22, 2026 at 9:11 AM James Lamb ***@***.***> wrote: *jameslamb* left a comment (scipy-conference/scipy_proceedings#1265) <#1265 (comment)> Also please do address the failing CI: No config file present for path "papers/SHAURYA_AGARWAL_Vogon_Poetry" Create papers/SHAURYA_AGARWAL_Vogon_Poetry/myst.yml and ensure it includes a valid project.id (build link <https://github.com/scipy-conference/scipy_proceedings/actions/runs/28449684178/job/84308240190?pr=1265> ) Also I think it's expected that you provide configuration for rendering the paper with MyST: https://github.com/scipy-conference/scipy_proceedings#write-your-paper — Reply to this email directly, view it on GitHub <#1265?email_source=notifications&email_token=AAA36GW7TZHCE2S5W3YFPW35GAZVTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBUGE2TQMRYGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5041582820>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA36GTZXMVX42XU2RDCVJ35GAZVTAVCNFSNUABDKJSXA33TNF2G64TZHMYTQMRVGYZDSO2JONZXKZJ3GQ3TONZTGQZDKMJVUF3AE> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AAA36GRPUSVL5ZOMIBSNAPD5GAZVTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBUGE2TQMRYGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AAA36GQXPQW3ZZ4QXK4FMRT5GAZVTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBUGE2TQMRYGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[fwkoch]

Hey @shauryashaurya - regarding MyST - I already made a comment on your other paper, same applies here.

I think this will be easier than you anticipate, though. You shouldn't need to change your markdown file at all (MyST takes normal markdown), only add the myst.yml metadata file (which you can just copy over from the template and edit).

I'm happy to help out if you need!

[fwkoch]

Regarding your concerns about deleting the SILMARIL files. This won't delete the whole PR. You can even delete them from the github web interface, from the Files changed tab:

[aditya14896]

Hi all, I'm Aditya Singh (@aditya14896). I'll be one of the reviewers for this paper - looking forward to reading it and will get a review posted soon.

[shauryashaurya]

Regarding your concerns about deleting the SILMARIL files. This won't delete the whole PR. You can even delete them from the github web interface, from the Files changed tab:

Thank you for the guidance! This worked - I have deleted the Silmaril paper from this branch! Woohoo!
Lesson learned :)

I think I am a bit more comfortable with MyST etc. - let me update that tomorrow (India time).

[sanhitamj]

@shauryashaurya

For this submission to be even reviewed properly, I will bring up the Review Criteria first:

• Written Quality:

  • Prose should clearly and consistently communicate the narrative (e.g. consistent tense and person; use of active voice; etc)
  • The written style should convey information that furthers the knowledge or research of the reader.
  • Due to the interdisciplinary nature of SciPy, highly domain-specific jargon should be avoided or explained where possible.

• Technical Content:

  • The technical content should be scientifically sound.
  • Computational content should, likewise, be verified.

Besides, lack of citation of others' work, lack of literature review also makes this submission not a reliable one.

Currently this submission reads more like a blog post than a scientific and/or technical paper. For any submission to be included in SciPy Proceedings, it should first pass the criteria above, and then through two reviewers' approval.