Advanced Grouping

[dplepage]

This branch implements a more powerful grouping mechanism, allowing the puzzle list to be grouped by any tag prefix or combination of prefixes. (palindrome issue #320).

The groupPuzzlesByTags function behaves like the previous puzzleGroupsByRelevance - it takes a list of puzzles and arranges them into nested groups for viewing. However, unlike its predecessor which only grouped things via the group: prefix and the administrivia tag, groupPuzzlesByTags takes a list of tag prefixes to group by, and thus supports re-grouping the same set of puzzles in different ways.

Motivation

2026 Kingdom of the Puzzmon

The motivating example for this was the Kingdom of the Puzzmon section of the 2026 MIT Mystery Hunt - this section had nine different regions of puzzles (each with its own page on the hunt website and its own minigame for unlocking puzzles in that region), as well as eight metapuzzles that each used some subset of the puzzles from the nine regions.

We found that we could not use JR to effectively visualize the assignments of puzzles to metas, because attempting to use the single group type for both region tags and artifact tags made the puzzle list extremely long and difficult to navigate.

General problem

The general problem is that when there are two (or more) orthogonal grouping mechanisms, using the same group: tag for everything breaks down for several reasons:

• Every puzzle is duplicated once for each group it belongs to, making the puzzle list extremely long
• The ordering intermixes groups of different types
• If the known puzzles in one group happen to all be within another, they'll be nested and then will unnest when a new puzzle is discovered.

E.g. given the following nine puzzle setup with two orthogonal categories:

JR's grouping produces a puzzle list like

• Region 1
  • Puzzle 1
  • Puzzle 2
  • Puzzle 3
• Meta 3
  • Puzzle 3
  • Puzzle 6
  • Puzzle 9
• Meta 1
  • Puzzle 1
  • Puzzle 4
  • Puzzle 7
• Region 2
  • Puzzle 4
  • Puzzle 5
  • Puzzle 6
• Region 3
  • Puzzle 7
  • Puzzle 8
  • Puzzle 9
• Meta 2
  • Puzzle 2
  • Puzzle 5
  • puzzle 8

The problem gets exponentially worse if there are more than two groupings - imagine a hypothetical hunt where puzzles are divided into rounds, correspond to different metas, AND all have one or more attributes attached to them (e.g. the way the 2018 emotions round assigned every puzzle one or more emotions - imagine if every puzzmon puzzle had had one or two pokemon types assigned to it, for example).

Grouping by tag prefix

The solution is to use semantically meaningful tags, and provide tools to group by any set of tags. In the Puzzmon Kingdom example, each puzzle would be tagged e.g. region:Aviaria and also meta:Chart your Course (or, better yet, artifact:Chart your Course, since the metapuzzles were called "artifacts" in that round).

Then you can group by region to see progress across the different regions, by artifact to see how puzzles are assigned to different artifacts, and by region, artifact or artifact, region to see nested groupings. What's more, new axes can be added simply by using another prefix - if all the puzzles have pokemon types, just tag them type:Fighting and type:Flying, and now can you group by those as well as needed.

Details

Grouping by multiple prefixes

When grouping by multiple prefixes, each one is applied independently to each previous group, so that if you group by region and then artifact you will see eight top-level groups (one per region), each of which is divided into up to nine subgroups (one per artifact); each subgroup will only contain the puzzles that belong to BOTH that artifact and that region.

Reversing the order of the grouping (i.e. artifact then region) will instead divide the puzzles up by artifact, and then within each artifact divide those puzzles up by region.

Automatic nesting

JR currently has a feature where if one group is detected to be a strict subset of another it will automatically be nested inside it. This is somewhat fragile - if someone adds a new puzzle and tags it group:Aviaria but forgets to tag it group:Kingdom of the Puzzmon, the shape of the board will drastically change as the entire Aviaria block ceases to be a subgroup of the Kingdom and gets promoted to be a top-level group.

This doesn't happen with semantically meaningful tags - if a puzzle is tagged region:Aviaria but not dimension:Kingdom of the Puzzmon, it will show outside of the Kingdom but it won't cause all the other Aviaria puzzles to be pulled out of place with it.

"None" groups

One thing I experimented with and would love feedback on is the automatic addition of "None" groups: there's a flag for groupPuzzlesByTags that makes it so that if there are puzzles with no relevant tag for a given prefix they will automatically display with e.g. region:None. I think this produces an overall more readable list, especially when there are several things you want to group by, but I'm not positive.

The currently-implemented behavior is a "None" section will be added if the flag is set and there is at least one grouped puzzle AND at least one ungrouped puzzle - if there are no ungrouped puzzles, it won't bother creating an empty group, and if there are no grouped puzzles it won't bother putting the entire list in a spurious "something:None" section (this makes it easy to have multiple groupings that don't all apply to every part of the hunt - e.g. if you're grouping the 2026 hunt by dimension, region you wouldn't want the other dimensions that don't have regions to all have spurious extra "region:none" groupings).

Merged groups

Another tweak I added is the ability for a grouping to have multiple shared tags - this allows merging groups that are identical, showing the common tags at the top.

meta-for:

The one nontrivial behavioral change this introduces is that currently the tag meta-for:X is understood to mean that this puzzle is the meta associated with group:X. Since the group prefix is no longer fixed, the code now expects instead to see the entire tag, e.g. meta-for:artifact:Chart your Course.

Administrivia

JR treats the administrivia tag like a group, but like any group if it's not a proper subset of another group it won't get rendered with that group. I implemented an administrivia-for: prefix that functions much like meta-for: in that it considers the puzzle to be part of whatever tag follows administrivia-for:, and automatically floats this puzzle to the top of the view for that category when you're viewing by that category.

There was also some existing logic in JR to sort puzzles with the administrivia tag but no group: tags above those with both administrivia and a group; I didn't keep that in because administrivia for a specific group should be marked administrivia-for instead of general administrivia.

Backwards compatibility

I left in the puzzleGroupsByRelevance function as a shorthand for groupPuzzlesByTags(["group"], nest=True, merge=False, makeNones=False); with these settings, the behavior is nearly identical to JR without this PR, except for a few small things:

• meta-for:foo now needs to be written meta-for:group:foo
• Puzzles tagged administrivia are now always shown at the top, even if all of them are tagged as part of another group.
• Puzzles tagged administrivia that are also tagged with group:<whatever> won't automatically sort lower in the administrivia section than puzzles that don't have assigned groups (but teams should use administrivia-for: instead to make admin notes for subgroups, so that it shows up in that subgroup)

Demo branch

This branch provides the code to group by arbitrary tag prefixes, but doesn't change the UI at all - it just invokes the more powerful grouping code with the single prefix "group" and flags to mimic the existing behavior.

In the dpl/groupby-demo branch, I added two quick-and-dirty hacks to better show off the grouping - first, there's a (very quick and dirty) "group by" options panel at the top of the puzzle list, so you can see what it looks like with different grouping strategies and with the automatic nesting, merging identical groups, and creation of "None" groups enabled or disabled, and second, there's a second available fixture, triggerable by calling Meteor.call("Hunts.methods.createGroupingDemo") from the browser console, that creates a proof-of-concept hunt that is the Puzzmon round from the 2026 hunt with every puzzle tagged with its region, its artifact, and two random pokemon-style types.

[zarvox]

First off: thanks for the detailed write-up -- it's always very helpful to fully appreciate the challenges or problems that people are facing, and I think I got a pretty good sense of what behaviors are frustrating. I tried out your dpl/groupby-demo branch to get a sense of what you were going for.

There's some valid criticism of what exists:

• meta-matching is tricky
• if a puzzle is missing a relevant tag then it can make the "magical" grouping not-so-magical and blow up the puzzle list page
• over-use of group: tags (especially orthogonal ones) blows up the size of the puzzle list page

(For what it's worth, Death and Mayhem did not use group: tags, or tags at all, for the Kingdom meta-matching this year; we did it all in the sheets for the metas.)

With that said, I'm different amounts of excited about different pieces of the ideas here. First, the good:

• I'm very supportive of collapsing groups with identical membership into a single group with multiple shared tags -- that seems like it'd make significantly better use of space and be clearer, and we should probably just do that all of the time, regardless. No need to make it a checkbox.

• I'd also love to reduce operator tagging error (and the confusing groups that can result from such errors), though I suspect we'll probably reach "good enough" with minimal complexity or rigidity in the data models by adding an "add new puzzle here" button in the group header that opens an add puzzle dialogue prepopulated with the corresponding group: tags. (As an operator for D&M, I would also love this as a labor-saving device.)

• I'm open to revisiting how administrivia behaves, though I'm not sure I'd reach for administrivia-for as the answer. I think the addition of administrivia-for seems maybe less about semantics/prioritization and more about wanting a way to sort something to the top of a particular context? I'm wondering if another approach to solving that problem space would be to have better affordances for sort order that operators could set on tags (perhaps via some sort of drag-and-drop tierlist-like UI)? I could see a world in which we might remove the special treatment of administrivia but allow sorting items or groups with particular tags

I'm less sold on displaying arbitrary hierarchies by facet, or "none" groups driving towards collectively-exhaustive tag categorization. I can see how they are potentially appealing from the perspective of wanting to categorize and add metadata to everything, but my experience in practice has been that we generally want data in jolly-roger to be things we are confident about, and often during hunt we often do not know exactly how things will behave, and to the extent that we have reasonable hypotheses, those are best worked on with adjacent discussion in a Google sheet, either for a specific puzzle or a stub puzzle (e.g. "Kingdom metas") where that work is being done.

Perhaps we could help track those sorts of things down with better filter-search, though: I can imagine solving the "what puzzles lack a tag?" with a "not" operator (specifying inversion of the match condition), and being able to specifically match on a particular field (e.g. title: or answer: or tag:). Meta-matching benefits from knowing what has and hasn't been used yet, so being able to tag things that have been matched and see just what hasn't seems handy. But I still think overall that's probably better done canonically in a "Kingdom metas" spreadsheet?

To your point about Region/Meta groupings being sorted in a seemingly-random order: today I would expect these to be sorted by their "level of doneness" -- the ones with solved metas should be lower in the list than the ones with unsolved metas. It's not obviously correct to me that matching prefixes should sort together. I could see a hypothetical tag-sort-order-override feature allowing producing this as an emergent property, though.

I also have some worries that if we make the puzzle list page a more per-user-stateful view than it is today, that we'll lose some of the cohesion of "everybody is usually looking at a curated view of the same data that is managed by the operators to surface relevant priorities." I'll grant that the existing unlock-order sort option technically risks that today, but I think that view probably gets minimal usage in practice.

I guess I don't have a great understanding of who this new view option/feature is for. It's probably not for people who just want to find something not-solved to work on. It's probably not sufficient for operators trying to optimize attention; to the extent that it will kinda-necessarily duplicate content that will be more-canonically worked on in sheets, it's at best a trailing indicator, so it's probably better for them to lurk in the relevant sheet to avoid multiple sources-of-truth and to understand what hypotheses about group assignment have what levels of confidence (and have all the usual programmatic tools for sorting/coloring/filtering/etc.). And while I can see how it could improve certain meta-matching tasks, I still think you'd be better off with a spreadsheet for most of that work, so I'm reticent to add additional complexity globally when the payoff seems low.

While I'm brainstorming: I can imagine a useful feature whereby jolly-roger propagates certain information about puzzles into a jolly-roger managed sheet/range of cells in the Google Sheet for related puzzles or something, to reduce the synchronization overhead, but I think that would also need some more design.

So with all that said, I think that there are some good ideas, some valid needs that I appreciate you highlighting but which I might like to try a different technical approach to, and some things where I think I'm unconvinced that the juice is worth the squeeze relative to just making a spreadsheet and using that. I'd like to take a more piece-by-piece approach here: I'd merge patches for:

• merging identical groups and displaying multiple shared tags as appropriate (I'd probably want some new unit test coverage for the changes to the grouping logic)
• an "add a puzzle" button in each related group header with the add puzzle modal prepopulated with the right set of tags

And I'd welcome some more discussion (perhaps in separate issues) around:

• tags carrying information about how puzzles with that tag (or groups sharing that tag) should be ordered to keep what's at the top of lists the most relevant
• more capable filter search

rather than needing a wholly different data hierarchy.

[dplepage]

Thanks for the detailed feedback!

Collapsing groups

I will definitely split this out into a separate PR; the checkbox in the UI was only so that I could see how looked either way, and I agree that if nobody objects it'd be better to just always do it.

Better filtering/searching

Huge +1 to this - I was actually already looking at incorporating https://github.com/projekt-apollo/query-parser because I think stronger filtering & searching would help a lot regardless of whether there's advanced grouping; the next time I have a chance to work on this I can pick up #1989 if nobody else already has, and look into liqe instead.

Administrivia-for

I only added this because there's some existing logic to make puzzles tagged administrivia without any group:* tags float above puzzles that have both administrivia AND a group:* tag. I have no particular attachment to this, I just didn't want to break it if anyone was currently relying on being able to mark administrivia for a specific group.

Operator error

Operator error has been more of a problem for Palindrome than for D&M, I think, because traditionally we haven't limited adding puzzles to a small dedicated team of operators. This means that it is actually fairly easy for someone to accidentally screw up the puzzle list for everyone, but possibly we should be trying to limit how many people add new puzzles rather than trying to make it harder to screw up.

Making the puzzle list too per-user-stateful

I didn't actually intend for the grouping to be customizable by everyone - I only threw in a UI component for that to make it easier to compare different options.

I think a better solution would be for there to be a hunt-level grouping, controlled by the operators, with non-operators only having the existing toggle to switch between "ungrouped" and "grouped". This would be sort of equivalent to @jimsug's proposed alternative of just having the operators mass-rename tags if we want to change the groupings, but without requiring a mass renaming - the operators could just go and declare a new group structure when appropriate.

Possibly it would also be useful to separate the "list of puzzles" page from the hunt "home" page? Then you could reorder, regroup, do complex searches, etc. however you want to on the "list of puzzles" page but there would still be a dedicated landing page that shows things in the canonical team-wide way?

That said, I think you've convinced me that the arbitrary group-by isn't an extremely useful feature; I'll see what I can do to extract the more useful parts of this PR into separate PRs.