[BUG] getWorkflowRunById reports one nested (sub-workflow) suspend as two suspended steps, so Studio renders two resume forms

[zouya8023-beep]

Describe the Bug

Upstream issue draft — nested (sub-workflow) suspend is reported as two suspended steps

Target: https://github.com/mastra-ai/mastra/issues/new

Status: draft, not submitted. Everything below was verified locally on 2026-08-11; the
"Verified / Not verified" markers are deliberate — please keep them honest if you edit.

---

Title

getWorkflowRunById reports one nested (sub-workflow) suspend as two suspended steps, so Studio renders two resume forms

---

Body

Summary

When a step inside a nested sub-workflow (e.g. a .dountil(...) loop used via .then(subWorkflow))
suspends, workflow.getWorkflowRunById(runId).steps contains two entries with
status: 'suspended' for that single suspension:

• the parent step entry — inner-loop
• the flattened nested entry — inner-loop.inner-step

snapshot.suspendedPaths is correct (one key). Only the steps map is doubled.

Because Studio renders one "Step suspended / Needs input" resume form per suspended step, a single
pending question can show up as two forms, and stale entries from an already-resumed nested step can
linger next to a genuinely-suspended later step.

There are three related data-quality problems in the same area, all reproduced by the same script
(details + repro output below):

1. Double reporting — one nested suspend → two status: 'suspended' entries.
2. suspendPayload / suspendedAt are retained after a step reaches success — so
   "does this step have a suspend payload?" is not a usable signal for "is it waiting on input?".
3. startedAt can be later than suspendedAt on a looped nested step, because a subsequent
   loop iteration overwrites startedAt while suspendedAt stays from the earlier iteration.
   Studio renders this as a negative duration.

Additionally, the two transports disagree on the identity of a nested step:

• REST / getWorkflowRunById → dotted key, inner-loop.inner-step
• stream / watch events on workflow.events.v2.${runId} → plain payload.id = 'inner-step'

Nested runs share the parent runId, so a parent-level watcher receives nested step events under the
plain id. Any client that merges "snapshot + live events" into one keyed map (Studio does) ends up
holding two identities for the same physical step, and a success event under one key does not clear
the suspended record under the other.

Reproduction

Self-contained script — a parent workflow with one nested dountil sub-workflow whose inner step
suspends once.

// repro.mjs   — deps: @mastra/core, @mastra/libsql, zod   ("type": "module")
import { rmSync } from 'node:fs';
import { z } from 'zod';
import { Mastra } from '@mastra/core';
import { createStep, createWorkflow } from '@mastra/core/workflows';
import { LibSQLStore } from '@mastra/libsql';

const DB = new URL('./repro.db', import.meta.url).pathname;
for (const s of ['', '-shm', '-wal']) rmSync(DB + s, { force: true });

const state = z.object({ round: z.number(), done: z.boolean() });

const inner = createStep({
  id: 'inner-step',
  inputSchema: state,
  outputSchema: state,
  suspendSchema: z.object({ question: z.string() }),
  resumeSchema: z.object({ answer: z.string() }),
  execute: async ({ inputData, resumeData, suspend }) => {
    if (resumeData) {
      // EXTRA_ITER=1: answering does not end the loop, so the loop re-enters and the
      // step body runs a second time (this is what triggers problem 3).
      return { round: inputData.round + 1, done: process.env.EXTRA_ITER !== '1' };
    }
    if (inputData.round > 0) return { round: inputData.round, done: true };
    return await suspend({ question: 'pick one' });
  },
});

const innerLoop = createWorkflow({ id: 'inner-loop', inputSchema: state, outputSchema: state })
  .dountil(inner, async ({ inputData }) => inputData.done)
  .commit();

const after = createStep({
  id: 'after-step',
  inputSchema: state,
  outputSchema: state,
  execute: async ({ inputData }) => inputData,
});

const parent = createWorkflow({ id: 'parent-wf', inputSchema: state, outputSchema: state })
  .then(innerLoop)
  .then(after)
  .commit();

const mastra = new Mastra({
  storage: new LibSQLStore({ id: 'repro', url: `file:${DB}` }),
  workflows: { parent },
  logger: false,
});
const wf = mastra.getWorkflow('parent');

function report(label, st) {
  const suspended = Object.entries(st.steps ?? {})
    .filter(([k, v]) => k !== 'input' && v?.status === 'suspended')
    .map(([k]) => k);
  console.log(`\n--- ${label} ---`);
  console.log(`run status     : ${st.status}`);
  console.log(`suspendedPaths : ${JSON.stringify(st.suspendedPaths ?? {})}`);
  console.log(`>>> steps with status==="suspended": ${suspended.length} ${JSON.stringify(suspended)}`);
  for (const [k, v] of Object.entries(st.steps ?? {})) {
    if (k === 'input' || !v || typeof v !== 'object') continue;
    console.log(
      `    ${k}: status=${v.status} startedAt=${v.startedAt} suspendedAt=${v.suspendedAt}` +
        ` resumedAt=${v.resumedAt} endedAt=${v.endedAt} hasSuspendPayload=${!!v.suspendPayload}`,
    );
    if (v.suspendedAt && v.startedAt && v.suspendedAt < v.startedAt) {
      console.log(`    >>> TIMESTAMP INVERSION on "${k}" (delta ${v.suspendedAt - v.startedAt}ms)`);
    }
  }
}

const run = await wf.createRun();
console.log(JSON.stringify(wf.serializedStepGraph?.map((e) => ({ type: e.type, id: e.id, stepId: e.step?.id }))));
await run.start({ inputData: { round: 0, done: false } });
report('after start — exactly ONE suspend happened', await wf.getWorkflowRunById(run.runId));

const rootStep = Object.keys((await wf.getWorkflowRunById(run.runId)).suspendedPaths ?? {})[0];
if (Number(process.env.DELAY_MS ?? 0) > 0) {
  await new Promise((r) => setTimeout(r, Number(process.env.DELAY_MS)));
}
// COLD=1 resumes via a fresh Run handle rebuilt from the snapshot, i.e. what an HTTP resume does.
const resumer = process.env.COLD === '1' ? await wf.createRun({ runId: run.runId }) : run;
await resumer.resume({ step: [rootStep, 'inner-step'], resumeData: { answer: 'a' } });
report('after resume — nothing should be suspended', await wf.getWorkflowRunById(run.runId));
process.exit(0);

Run:

node repro.mjs                                        # problems 1 and 2
COLD=1 DELAY_MS=3000 EXTRA_ITER=1 node repro.mjs      # additionally problem 3

Actual output (@mastra/core@1.57.0)

--- after start — exactly ONE suspend happened ---
run status     : suspended
suspendedPaths : {"inner-loop":[0]}
>>> steps with status==="suspended": 2 ["inner-loop","inner-loop.inner-step"]
    inner-loop:            status=suspended ... hasSuspendPayload=true
    inner-loop.inner-step: status=suspended ... hasSuspendPayload=true

--- after resume — nothing should be suspended ---
run status     : success
>>> steps with status==="suspended": 0 []
    inner-loop:            status=success ... suspendedAt=...300 hasSuspendPayload=true   # payload retained
    inner-loop.inner-step: status=success ... suspendedAt=...300 hasSuspendPayload=true   # payload retained

With COLD=1 DELAY_MS=3000 EXTRA_ITER=1:

    inner-loop.inner-step: status=success startedAt=1786427069305 suspendedAt=1786427066300 ...
    >>> TIMESTAMP INVERSION on "inner-loop.inner-step" (delta -3005ms)

Expected

• steps should identify a nested step once, under one stable key, and only the step that is
  actually awaiting input should carry status: 'suspended' (the parent container entry should not
  duplicate it).
• The same step should have the same identity in getWorkflowRunById and in the
  workflow.events.v2.* watch stream, so a UI can merge snapshot + live events without duplicates.
• suspendPayload / suspendedAt should be cleared (or clearly scoped to the current attempt) once
  a step reaches a terminal status.
• startedAt <= suspendedAt should hold for whatever attempt the record describes.

Environment

@mastra/core	1.57.0 (also reproduced on 1.50.1, see below)
mastra (CLI + Studio)	1.23.0
@mastra/libsql	1.19.0
Node	25.3.0
OS	macOS 14.4.1 (darwin, arm64)
storage	LibSQLStore, local file

Is this a regression?

No — verified. I ran the identical script against @mastra/core@1.50.1 + @mastra/libsql@1.15.1
and against 1.57.0 + 1.19.0. Problems 1, 2 and 3 reproduce identically in both. The dotted
`${step}.${key}` merge and the plain-step.id stream payloads exist in 1.50.1 too.

The one thing that did change in this area is the serialized graph shape for nested
sub-workflows (documented under 1.56.0, "Nested workflows as a first-class serialized step type"):

1.50.1:  [{"type":"step","stepId":"inner-loop"},   {"type":"step","stepId":"after-step"}]
1.57.0:  [{"type":"workflow","id":"inner-loop"},   {"type":"step","stepId":"after-step"}]

So the underlying data problems are long-standing; what changed for us is that after upgrading
(mastra 1.18.2 → 1.23.0, core 1.50.1 → 1.57.0) the Studio UI started surfacing them. Concretely, in
Studio we now see, on a real run with one nested suspend:

• the resume form titled with the dotted id aisearch-clarify-loop.aisearch-clarify
• a Timeline listing both Aisearch Clarify Loop and Aisearch Clarify Loop Aisearch Clarify
• a negative duration on the nested entry: Aisearch Clarify Loop Aisearch Clarify -0.001s
  (and 1190000s on its parent)

Not verified

We also observed, on a long-running real workflow, two resume forms at once (one for an
already-answered nested clarify step, one for a genuinely-suspended later step) plus an
out-of-order / incomplete node list — but only in a browser tab kept open across a resume; a hard
reload always renders correctly. We could not capture that client state before it was lost, so we are
not claiming a specific Studio-side cause here. The four data problems above are what we can prove,
and they are sufficient to explain a client that merges snapshot + stream into one keyed map ending
up with stale suspended records.

---

Local notes (not part of the issue)

• Working repro harness: data/repro-nested/ (gitignored). old/ and new/ are sibling sandboxes
  whose node_modules/@mastra/{core,libsql} symlink into node_modules/.pnpm/... for the two
  version pairs, so both can run the same repro.mjs. Recreate with the ln -s block from the
  session, or delete the directory — nothing else depends on it.
• Our own affected code: getWorkflowState() in
  src/mastra/common/libs/workflow/run-snapshot.ts merges every context[*].output in insertion
  order, and the dotted nested entries now land after their parent. On the run we inspected the
  parent and nested outputs were byte-identical, so nothing broke — but if a loop's last inner output
  ever diverges from the loop output, the dotted entry would silently win. Worth a guard that skips
  keys containing ..

Steps To Reproduce

// repro.mjs — deps: @mastra/core, @mastra/libsql, zod ("type": "module")
import { rmSync } from 'node:fs';
import { z } from 'zod';
import { Mastra } from '@mastra/core';
import { createStep, createWorkflow } from '@mastra/core/workflows';
import { LibSQLStore } from '@mastra/libsql';

const DB = new URL('./repro.db', import.meta.url).pathname;
for (const s of ['', '-shm', '-wal']) rmSync(DB + s, { force: true });

const state = z.object({ round: z.number(), done: z.boolean() });

const inner = createStep({
id: 'inner-step',
inputSchema: state,
outputSchema: state,
suspendSchema: z.object({ question: z.string() }),
resumeSchema: z.object({ answer: z.string() }),
execute: async ({ inputData, resumeData, suspend }) => {
if (resumeData) {
// EXTRA_ITER=1: answering does not end the loop, so the loop re-enters and the
// step body runs a second time (this is what triggers problem 3).
return { round: inputData.round + 1, done: process.env.EXTRA_ITER !== '1' };
}
if (inputData.round > 0) return { round: inputData.round, done: true };
return await suspend({ question: 'pick one' });
},
});

const innerLoop = createWorkflow({ id: 'inner-loop', inputSchema: state, outputSchema: state })
.dountil(inner, async ({ inputData }) => inputData.done)
.commit();

const after = createStep({
id: 'after-step',
inputSchema: state,
outputSchema: state,
execute: async ({ inputData }) => inputData,
});

const parent = createWorkflow({ id: 'parent-wf', inputSchema: state, outputSchema: state })
.then(innerLoop)
.then(after)
.commit();

const mastra = new Mastra({
storage: new LibSQLStore({ id: 'repro', url: file:${DB} }),
workflows: { parent },
logger: false,
});
const wf = mastra.getWorkflow('parent');

function report(label, st) {
const suspended = Object.entries(st.steps ?? {})
.filter(([k, v]) => k !== 'input' && v?.status === 'suspended')
.map(([k]) => k);
console.log(\n--- ${label} ---);
console.log(run status : ${st.status});
console.log(suspendedPaths : ${JSON.stringify(st.suspendedPaths ?? {})});
console.log(>>> steps with status==="suspended": ${suspended.length} ${JSON.stringify(suspended)});
for (const [k, v] of Object.entries(st.steps ?? {})) {
if (k === 'input' || !v || typeof v !== 'object') continue;
console.log(
${k}: status=${v.status} startedAt=${v.startedAt} suspendedAt=${v.suspendedAt} +
resumedAt=${v.resumedAt} endedAt=${v.endedAt} hasSuspendPayload=${!!v.suspendPayload},
);
if (v.suspendedAt && v.startedAt && v.suspendedAt < v.startedAt) {
console.log( >>> TIMESTAMP INVERSION on "${k}" (delta ${v.suspendedAt - v.startedAt}ms));
}
}
}

const run = await wf.createRun();
console.log(JSON.stringify(wf.serializedStepGraph?.map((e) => ({ type: e.type, id: e.id, stepId: e.step?.id }))));
await run.start({ inputData: { round: 0, done: false } });
report('after start — exactly ONE suspend happened', await wf.getWorkflowRunById(run.runId));

const rootStep = Object.keys((await wf.getWorkflowRunById(run.runId)).suspendedPaths ?? {})[0];
if (Number(process.env.DELAY_MS ?? 0) > 0) {
await new Promise((r) => setTimeout(r, Number(process.env.DELAY_MS)));
}
// COLD=1 resumes via a fresh Run handle rebuilt from the snapshot, i.e. what an HTTP resume does.
const resumer = process.env.COLD === '1' ? await wf.createRun({ runId: run.runId }) : run;
await resumer.resume({ step: [rootStep, 'inner-step'], resumeData: { answer: 'a' } });
report('after resume — nothing should be suspended', await wf.getWorkflowRunById(run.runId));
process.exit(0);

Link to Minimal Reproducible Example

https://github.com/mastra-ai/mastra/issues/new?template=bug_report.yml

Expected Behavior

no

Environment Information

no

Verification

• I have searched the existing issues to make sure this is not a duplicate
• I have included sufficient information for the team to reproduce and understand the issue

[daneatmastra]

Thank you for reporting this issue! We have labeled it for trio-tnt and we will look into it as soon as possible.

If you're experiencing an error, please provide a minimal reproducible example whenever possible to help us resolve it quickly.

[mastra-platform]

Type: bug — Genuine data-quality bug in nested-workflow run state: one nested suspend surfaces as two suspended step entries, plus stale suspendPayload/suspendedAt and inverted timestamps.
Route: Plan fix
Severity: 🟠 high — HITL resume in Studio is the affected surface; duplicated/stale resume forms make nested suspend/resume confusing and error-prone, though a workaround exists (resume via suspendedPaths).
Confidence: high — every reported symptom maps to specific code in packages/core/src/workflows.
Next step: Plan a fix in Workflow.getWorkflowRunSteps (dedupe container vs. flattened nested suspends), in packages/core/src/workflows/handlers/step.ts (clear suspend fields on terminal/re-entry), and align nested step identity between snapshot and watch stream.

Understanding

All four reported problems are real and reproduce from code inspection of main.

1. Double suspended entry. Workflow.getWorkflowRunSteps (packages/core/src/workflows/workflow.ts:3015-3044) copies each snapshot step and, when the entry is a nested workflow, additionally merges the child run's steps under `${step}.${key}`. The parent container entry is itself recorded as suspended with an aggregated suspendPayload (built in packages/core/src/workflows/default.ts:643-655), so one physical suspend yields both inner-loop and inner-loop.inner-step with status: 'suspended'. suspendedPaths stays correct (one key) because it is written independently. Studio derives resume forms directly from this map — useSuspendedSteps in packages/playground/src/domains/workflows/workflow/use-workflow-trigger.ts:31-42 filters steps by status === 'suspended' with no dedupe or path awareness, and workflow-suspended-steps.tsx renders one form per entry.

2/3. Stale suspendPayload / suspendedAt, and startedAt > suspendedAt. packages/core/src/workflows/handlers/step.ts:159-166 builds stepInfo by spreading the previous stepResults[step.id], then only overwrites payload/startedAt/resumedAt/status. The terminal branch at step.ts:502-508 sets status: 'success' and endedAt but never deletes suspendPayload/suspendedAt, so they survive into the terminal record. In a loop, the next iteration writes a fresh startedAt while the stale suspendedAt from the earlier iteration persists — exactly the negative duration the reporter observed. Both are the same root cause: suspend-scoped fields are not reset on step re-entry/termination.

4. Identity mismatch between snapshot and watch stream. A nested workflow run is created with the parent's runId (workflow.ts:2773-2775), so nested step events published to workflow.events.v2.${runId} are picked up unprefixed by the parent's own subscriber, while the nested-watch relay republishes the same events with an `${workflowId}.${id}` prefix (workflow.ts:4082-4110). A client merging snapshot + live events therefore holds two identities for one step, and a success event under one key never clears the suspended record under the other. Secondary hazard: the relay prefixes with the nested workflow id, whereas the snapshot flatten prefixes with the parent step key — these diverge for cloned/aliased sub-workflows.

Suggested direction. (a) In getWorkflowRunSteps, keep one authoritative identity for a nested step and don't let the container entry report suspended when a flattened child entry already does (or have Studio derive forms from suspendedPaths); (b) reset suspendPayload/suspendedAt/suspendOutput in step.ts when a step re-enters or reaches a terminal status; (c) unify nested step identity across REST and watch transports. Related open issues in the same area (not duplicates): #20660, #19396.

Assumptions

• The reporter's environment (@mastra/core@1.57.0) is close enough to main that the traced code paths are current; no commit reverting this behavior was found.
• Studio is the primary affected surface; the underlying defect is in core, so the fix belongs in core rather than in the playground filter.
• Marking the nested-workflow container entry suspended is intentional (it carries the aggregated payload used for [parent, child] resume targeting), so the fix should disambiguate for consumers rather than stop recording it.
• Problem 4 (transport identity mismatch) is treated as part of the same defect cluster because it produces the same duplicate-form symptom.
• The reporter's aside about dotted-key collisions overwriting a loop output is a plausible latent hazard but was not independently verified; it is a guard worth adding, not a confirmed bug.

Open questions

• Whether the container (inner-loop) entry should keep status: 'suspended' in the public steps map is an API-shape decision: changing it is arguably a breaking change for consumers already keying off it.
• Which identity should be canonical for nested steps — the parent step key (inner-loop.inner-step) or the nested workflow id — given cloned sub-workflows make them diverge.