Skip to content

Route hosts to instance groups from an inventory variable#568

Open
fernandorocagonzalez wants to merge 1 commit into
ctrliq:mainfrom
fernandorocagonzalez:feat-ig-routing
Open

Route hosts to instance groups from an inventory variable#568
fernandorocagonzalez wants to merge 1 commit into
ctrliq:mainfrom
fernandorocagonzalez:feat-ig-routing

Conversation

@fernandorocagonzalez

Copy link
Copy Markdown
Contributor

Implements the RFE in #566: route hosts to instance groups based on an inventory variable.

The problem

With several isolated network zones, each zone has its own execution nodes in an instance group because firewalls only let those nodes reach the hosts of their zone. A job runs on one instance group, so today every baseline template (sysctl, users, patching) has to be duplicated per zone with a different limit and instance group. N templates and M zones means N x M copies that only differ in two fields.

What this does

New opt in field on job templates, instance_group_routing_var: the name of an inventory variable (in practice usually a group var) whose value names the instance group each host should run on.

# group_vars of datacenter1
dc_instance_group: dc1-nodes
# group_vars of datacenter2
dc_instance_group: dc2-nodes

At launch, hosts are bucketed by the resolved value and the launch creates the same implicit workflow job that sliced jobs use, one node per bucket. Each job runs restricted to the hosts of its bucket (filtered in the generated inventory, like slicing filters each slice, so limit keeps working) and assigned to the instance group named by the value. Hosts that do not resolve the variable run in one extra job with the normal instance group selection. If everything lands in a single bucket, a plain job is launched on it, and if nothing resolves the variable the launch behaves as if routing were not configured.

Resolution follows a simplified version of the Ansible precedence rules: host vars win over group vars, groups merge sorted by depth, then ansible_group_priority, then name. Only enabled hosts count. Details in the new docs/instance_group_routing.md.

Federated inventories in this repo already do per source inventory instance group routing with the same implicit workflow pattern; this is the per host variable version of that idea for setups where all the zones live in one inventory.

Security

The routing variable is inventory data, editable by people who may have no permission at all on the instance groups, so it is not trusted blindly:

  • Routing only happens when the template explicitly opts in.
  • The launch endpoint checks the launching user's use permission on every referenced instance group, the same use_role rule that already applies to instance groups prompted at launch, and rejects the launch naming the group when the check fails or the group does not exist. No silent fallback. The validated buckets are handed to the job creation so the launch routes to exactly what was checked.
  • Relaunching a routed workflow recomputes the buckets and applies the same checks against the relaunching user. Relaunching a routed child job keeps targeting its original bucket (the routed group comes back through the launch config, which already carries the usual relaunch RBAC check).
  • Launch paths without a requesting user (schedules, webhooks, workflow nodes) cannot check a user's roles. They stay loud but safe when a value names a missing group: a workflow containing the routed template is marked failed with the reason in its explanation instead of crashing the workflow manager cycle, webhook launches answer 400, and scheduled launches are skipped with the error in the dispatcher log. If you want a stronger guarantee for those paths I am happy to iterate on it here, it is the main open question of the RFE.

Interactions

  • Mutually exclusive with job slicing: validation error on the template, and the launch endpoint rejects a prompted slice count above 1.
  • Prompting instance groups at launch overrides routing for that launch.
  • Provisioning callbacks never route, the same way they never slice: one calling host needs no fan out.

Testing

  • 21 new functional tests: variable resolution precedence (host vs group, depth, priority, disabled hosts), bucketing and node spawning, single bucket, unknown groups, RBAC on launch and relaunch, prompted overrides, relaunch after template edits, and a task manager test proving a routed node with a missing group fails the workflow instead of wedging the scheduler.
  • Jest test for the new form field.
  • flake8, black and eslint clean on the touched files.

Note: the migration is numbered 0201 and will collide with the other open PRs that also add 0201 (#561, #563, #564, #565); whichever merges second renumbers, same as discussed there.

In deployments with several isolated network zones, each zone has its
own execution nodes grouped in an instance group, because firewalls only
let those nodes reach the hosts of their zone. A job runs on a single
instance group, so the same baseline template (sysctl, users, patching)
has to be duplicated per zone with a different limit and instance group.

This adds an instance_group_routing_var field to job templates: the name
of an inventory variable (usually a group var) whose value names the
instance group each host should run on. At launch, hosts are grouped by
the resolved value and the launch creates the same implicit workflow job
that sliced jobs use, with one node per bucket: each job runs restricted
to its bucket's hosts and assigned to the routed instance group. Hosts
that do not resolve the variable run in an extra job with the normal
instance group selection, and if everything lands in one bucket a plain
job is launched on it. Resolution follows a simplified version of the
Ansible precedence rules (host vars win, then groups by depth, priority
and name) and only considers enabled hosts.

Instance groups are protected by RBAC and the routing variable is
inventory data, so it is not trusted blindly: the launch endpoint checks
the launching user's use permission on every referenced instance group,
the same rule already applied to instance groups prompted at launch, and
unknown names reject the launch instead of falling back silently. The
validated buckets are handed to the job creation so the launch routes to
exactly what was checked. The same checks run again when relaunching a
routed workflow, which recomputes the buckets from current inventory
data. Routing is mutually exclusive with job slicing (also when the
slice count is prompted at launch), prompting instance groups overrides
it, and provisioning callbacks never route, like they never slice.

Launch paths without a requesting user stay loud but safe when a routing
value names a missing instance group: a workflow containing the routed
template is marked failed with the reason instead of crashing the
workflow manager cycle, webhook launches answer 400, and relaunching a
routed child job keeps targeting its original bucket even if the
template's routing variable changed since.

Refs ctrliq#566
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants