Add automatic workflow node retries via a max_retries node setting

[fernandorocagonzalez]

Summary

Workflow nodes get a max_retries setting (0 to 100, default 0): when the node's job fails, the scheduler respawns it automatically up to that many times before following the node's failure paths.

max_retries: 2  ->  up to 3 executions: the original attempt plus 2 retries

Each node keeps its own independent budget, so one flaky node does not eat the retries of its siblings. This covers the classic "the playbook failed because the network hiccuped, run it again" case that today means either failing the whole branch or relaunching by hand. It complements the relaunch-from-failed feature: retries are automatic and happen inside the run, relaunch is manual and happens after it. Relaunched workflows start their nodes with fresh counters.

Earlier revisions of this PR drove this through a reserved ascender_max_retries extra var. Following review feedback it is now a proper field on the workflow node, which killed all the variable plumbing: no value parsing or clamping, no stripping the var before it reaches playbooks and nested or sliced workflows, no serializer exemption for unprompted vars. The field is validated at the API boundary (0 to 100, capped because there is no delay between attempts) and is editable from the node form in the visualizer.

How it behaves

• max_retries lives on the workflow job template node and is copied to the workflow job node at launch, like all_parents_must_converge, so editing the template does not affect running workflows. Copying a workflow job template preserves it
• While a node has retries left, a failed or errored job does not send the workflow down the failure paths. The scheduler treats the node as if its job were still running: children stay undecided, convergence nodes keep waiting, the workflow stays running. On the next cycle a new job is spawned with the same launch configuration and the node's read only retry_attempts counter goes up
• The node's job field always points at the newest attempt. Superseded attempts are kept in a new retried_jobs relation on the node, so they stay linked to their workflow (source_workflow_job still resolves) and cannot be deleted while the workflow is running
• Only when the budget is spent does the node count as failed, and its failure and always paths run against the last attempt
• Some failures are never retried because retrying cannot change the outcome: canceled jobs (that was a human pressing cancel), approval nodes (also a human decision), nodes whose template was deleted mid run, and jobs that could not start at all (missing project or inventory, workflow recursion), which exhaust the budget immediately instead of hot looping
• A nested workflow node retries the inner workflow as a whole; nodes inside it retry only if its own template says so. Slices of a sliced job template are not retried individually

Design notes

• The "is this node really done" logic lives in three small methods on WorkflowJobNode (retry_pending(), job_finished(), finally_failed()) and the DAG predicates in dag_workflow.py consume those instead of each keeping its own status list, so the invariant has one home
• The respawn itself is a few lines in spawn_workflow_graph_jobs: a node coming back from bfs_nodes_to_run() with a job attached is a retry, so the old job goes into retried_jobs, the counter increments and a new unified job is created exactly like the first attempt (fresh ancestor artifacts included)
• The approval exclusion resolves the job class through the polymorphic ctype rather than isinstance, so it keeps working even if someone later adds select_related('job') to the scheduler queryset (that returns base UnifiedJob instances, there is already a warning about it in the file)
• One migration (0203, depends on 0202_project_webhooks) adding max_retries to both node tables plus retry_attempts and the retried_jobs table on the job node
• UI: a Max Retries number input in the visualizer node form (hidden for approval nodes) and the value shown in the node details view. awxkit passes the field through
• Docs: new "Automatic Node Retries" section in docs/workflow.md

Tests

• Unit: the three node helpers and never-retried cases (canceled, approvals, deleted template, exhausted budget), DAG behavior (respawn instead of failure path, exhausted budget follows failure paths, convergence nodes wait for a retrying parent, a retry-pending leaf does not fail the workflow), and the template-to-job-node field copy
• Functional: DAG runs with real models (retry then exhaustion, per node budgets, deleted template finishes the workflow as failed), API validation of the field including the 0 to 100 bounds, deletion protection of superseded attempts, and two full lifecycle tests driving TaskManager plus WorkflowManager: fail, respawn, succeed, workflow successful; and fail, respawn, fail again, workflow failed with exactly two jobs
• UI: jest coverage of the new form field default through the node modals, reducer and save payloads

[Copilot]

Pull request overview

Adds automatic per-node workflow retries controlled by a reserved variable (ascender_max_retries), integrating the retry semantics into the DAG scheduler and ensuring retried attempts remain linked/protected for traceability and deletion safety.

Changes:

• Introduces WorkflowJobNode retry state (retry_attempts, retried_jobs) and scheduler logic to respawn failed/error nodes while budget remains.
• Strips ascender_max_retries from spawned job extra_vars and relaxes node serializer prompt-validation to allow the reserved var in extra_data.
• Adds unit/functional coverage and documents the new “Automatic Node Retries” behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/workflow.md	Documents reserved-var configuration, semantics, and exclusions for automatic node retries.
awx/main/tests/unit/scheduler/test_dag_workflow.py	Unit tests for DAG behavior when retries are pending/exhausted (including convergence blocking).
awx/main/tests/unit/models/test_workflow_unit.py	Unit tests for retry budget resolution/precedence and helper methods (retry_pending, etc.).
awx/main/tests/functional/task_management/test_scheduler.py	End-to-end TaskManager/WorkflowManager lifecycle tests for respawn and exhaustion.
awx/main/tests/functional/models/test_workflow.py	Functional DAG tests validating respawn behavior, template deletion, and per-node overrides.
awx/main/tests/functional/api/test_workflow_node.py	Functional API tests for reserved-var acceptance and deletion protection of superseded attempts.
awx/main/scheduler/task_manager.py	Implements retry respawn mechanics, increments counters, and tracks superseded jobs.
awx/main/scheduler/dag_workflow.py	Centralizes “node finished/final failure” logic via WorkflowJobNode helper methods.
awx/main/models/workflow.py	Adds retry fields/constants and implements max-retry resolution + helper predicates + var stripping.
awx/main/models/unified_jobs.py	Preserves workflow linkage for superseded attempts via retried_workflow_nodes.
awx/main/migrations/0201_workflowjobnode_retry_attempts.py	Migration adding retry_attempts and retried_jobs to WorkflowJobNode.
awx/api/views/mixin.py	Prevents deletion of superseded attempts while their workflow job is still active.
awx/api/serializers.py	Allows reserved retry var through node prompt validation; exposes retry_attempts/retried_jobs.

[cigamit]

Needs rebasing and fixing the migration name/dependency, then I will review.

But also a thought, why use an extra var instead of creating an actual workflow setting? I really dislike relying on user supplied ansible variables for things like this.

[fernandorocagonzalez]

Rebased onto main and renumbered the migration to 0203 with its dependency on 0202_project_webhooks.

On the extra var concern: you were right, and it turned out to be less work than expected because the variable plumbing was most of the complexity. max_retries is now a real field on the workflow node (template and job node, copied at launch like all_parents_must_converge). That removed the value parsing and clamping, the strip that kept the var away from playbooks and nested/sliced workflows, and the serializer exemption for unprompted vars. Validation is now plain DRF (0 to 100).

What it adds instead: the field in both node serializers, awxkit passthrough, and a Max Retries number input in the visualizer node form (hidden for approval nodes since those never retry) plus the value in the node details view.

I squashed everything into one commit since the old ascender_max_retries history no longer made sense, and updated the PR description. Backend suites (workflow unit, functional models/api/scheduler), jest on the touched screens, flake8/black and makemigrations --check are all green. Ready for review.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.