fix(llmobs): pre-parse OpenAI AsyncAPIResponse to fix async streaming span lifecycle [MLOB-6778] by heyitsgrace996 · Pull Request #17432 · DataDog/dd-trace-py

heyitsgrace996 · 2026-04-09T16:02:47Z

Description

Cherry-pick of #17061 (authored by @cedomir) with additional safety improvements and E2E regression tests.

Fixes #17060 — when using the OpenAI Agents SDK with Runner.run_streamed(), streaming spans for responses.create(stream=True) are never finished, breaking LLM Observability trace grouping for sub-agent calls.

Root cause

_EndpointHook.handle_request (a sync generator) calls resp.parse() on AsyncAPIResponse. Since parse() is async def, this returns an unawaited coroutine. The coroutine flows into _handle_streamed_response, which can't recognize it as an AsyncStream, so the stream is never wrapped in TracedAsyncStream. The span is never finished, corrupting parent context for subsequent sub-agent calls.

Fix

Pre-parse AsyncAPIResponse via _maybe_preparse_async_response() before sending into the sync _traced_endpoint generator, so the hook receives the actual AsyncStream.
Inject the hook's TracedAsyncStream back into the response's parse cache via _inject_into_parse_cache() so the SDK caller iterates the traced version — which finishes the span on stream completion.

Additional changes (beyond cherry-pick)

Safety wrap: _inject_into_parse_cache() now wrapped in try/except Exception to prevent tracing from raising into application code if SDK private internals change. Matches the safety pattern already used in _maybe_preparse_async_response().
E2E regression tests: 3 new tests in test_openai_v1.py using VCR cassettes and test_spans assertions:
- test_chat_completion_async_stream_span_finished — Chat Completions async streaming
- test_response_async_stream_span_finished — Responses API async streaming
- test_achat_completion_raw_response_non_stream — Async non-streamed with_raw_response (parse cache path)
Lint fix: Removed unused import asyncio from unit test file.

Before / After

Scenario	Before	After
`Runner.run_streamed()` with sub-agents	Orphaned spans, `RuntimeWarning: coroutine 'AsyncAPIResponse.parse' was never awaited`, spans dropped at shutdown	All spans properly grouped under workflow, clean exit
`Runner.run()` (non-streaming)	Works correctly	No change
Non-async streaming	Not affected	Not affected

Before (unpatched ddtrace 4.6.7):

After (patched ddtrace 4.8.0rc2):

Testing

Unit tests for _inject_into_parse_cache (SDK v1.x and v2.x) and _maybe_preparse_async_response (async/sync parse, error safety, None handling)
E2E regression tests: async streaming span lifecycle for Chat Completions and Responses API (VCR cassettes, parent-child span assertions, span.duration is not None)
E2E regression test: async non-streamed with_raw_response parse cache path
Full openai integration suite: 181 passed, 23 skipped on Python 3.12
Manual repro with OpenAI Agents SDK (Runner.run_streamed() with sub-agent handoffs) — confirmed broken behavior on ddtrace 4.6.7, confirmed fix on 4.8.0rc2
Lint: ruff, mypy, bandit clean (2 expected low-severity B110 warnings for intentional try/except/pass)

Risks

Accesses OpenAI SDK private internals (_parsed_by_type for v2.x, _parsed for v1.x) for parse cache injection — could break on SDK updates. Both access points wrapped in try/except for safety; failure degrades to pre-fix behavior (unfinished spans), never crashes the application.
parse() is now called eagerly (before the hook) rather than lazily. OpenAI SDK's parse() is idempotent/cacheable, so no side-effect risk with the current SDK.

…ming span lifecycle _EndpointHook.handle_request (a sync generator) calls resp.parse() on AsyncAPIResponse, which returns an unawaited coroutine instead of the actual stream. This prevents the stream from being wrapped in TracedAsyncStream, so the span for responses.create(stream=True) is never finished — breaking LLMObs trace grouping for streamed sub-agent calls (e.g. via OpenAI Agents SDK's Runner.run_streamed()). The fix: 1. Pre-parse AsyncAPIResponse in async_wrapper and _trace_and_await before sending into the sync _traced_endpoint generator, so the hook receives the actual AsyncStream and can wrap it. 2. Inject the hook's TracedAsyncStream back into the response's parse cache (_parsed_by_type for SDK v2.x, _parsed for v1.x) so callers iterate the traced version, which finishes the span on completion. Fixes #17060 Made-with: Cursor

…e tests - Extract `_maybe_preparse_async_response()` to deduplicate pre-parse logic between `async_wrapper` and `_trace_and_await` (Kyle comment on duplication) - Wrap pre-parse in try/except so parse failures never suppress the original exception or change application-visible behaviour (Kyle + Codex comments) - Add SDK version documentation to `_inject_into_parse_cache` docstring noting which private internals correspond to which SDK versions - Unify post-hook injection logic between `_trace_and_await` and `async_wrapper` for consistency (Kyle asymmetry comment) - Rewrite tests: replace class-based MagicMock tests with parametrized functions using typed fake objects; add comprehensive tests for the new `_maybe_preparse_async_response` helper including error safety; remove broken FakeEndpointHook test (Codex comment) Made-with: Cursor

Wrap _inject_into_parse_cache in try/except to prevent tracing from raising into application code if OpenAI SDK private internals change. Add two E2E regression tests validating that async streaming spans are properly finished with correct parent-child relationships, covering both Chat Completions and Responses API paths. Addresses review feedback from PR #17061.

cit-pr-commenter-54b7da · 2026-04-09T16:03:29Z

Codeowners resolved as

ddtrace/contrib/internal/openai/patch.py                                @DataDog/ml-observability
releasenotes/notes/fix-openai-async-streaming-span-lifecycle-bdf04be937e24f07.yaml  @DataDog/apm-python
tests/contrib/openai/test_openai_v1.py                                  @DataDog/ml-observability

heyitsgrace996 · 2026-04-09T16:09:07Z

@codex

chatgpt-codex-connector · 2026-04-09T16:15:17Z

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…B-6778]

emmettbutler

release note looks fine

ncybul

Left a couple comments / questions, but the overall structure looks good to me!

…x [MLOB-6778] E2E regression tests in test_openai_v1.py already cover these paths.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f7c73e2f1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f7c73e2f1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…LOB-6778]

Cedo Segulja and others added 4 commits April 9, 2026 10:39

fix(openai): remove unused asyncio import from async streaming tests

f239648

heyitsgrace996 force-pushed the grace.williams/fix-openai-async-streaming-span branch from f6de1e0 to 0271a43 Compare April 9, 2026 16:11

heyitsgrace996 force-pushed the grace.williams/fix-openai-async-streaming-span branch 2 times, most recently from c3c4ff4 to 3c587bf Compare April 9, 2026 17:11

heyitsgrace996 marked this pull request as ready for review April 9, 2026 17:18

heyitsgrace996 requested review from a team as code owners April 9, 2026 17:18

heyitsgrace996 requested review from christophe-papazian and ncybul April 9, 2026 17:18

heyitsgrace996 changed the title ~~fix(openai): pre-parse AsyncAPIResponse to fix async streaming span lifecycle~~ fix(openai): pre-parse AsyncAPIResponse to fix async streaming span lifecycle [MLOB-6778] Apr 9, 2026

heyitsgrace996 changed the title ~~fix(openai): pre-parse AsyncAPIResponse to fix async streaming span lifecycle [MLOB-6778]~~ fix(llmobs): pre-parse OpenAI AsyncAPIResponse to fix async streaming span lifecycle [MLOB-6778] Apr 9, 2026

heyitsgrace996 mentioned this pull request Apr 9, 2026

fix(openai): pre-parse AsyncAPIResponse in async wrapper to fix streaming span lifecycle #17061

Closed

5 tasks

fix(openai): add async non-streamed raw-response regression test [MLO…

6415de8

…B-6778]

emmettbutler approved these changes Apr 9, 2026

View reviewed changes

ncybul reviewed Apr 9, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/openai/patch.py

Comment thread tests/contrib/openai/test_openai_async_streaming.py Outdated

fix(llmobs): remove additional unit tests for async streaming span fi…

0824d86

…x [MLOB-6778] E2E regression tests in test_openai_v1.py already cover these paths.

heyitsgrace996 force-pushed the grace.williams/fix-openai-async-streaming-span branch from 3c587bf to 0824d86 Compare April 10, 2026 18:01

Merge branch 'main' into grace.williams/fix-openai-async-streaming-span

7c3c58e

ncybul approved these changes Apr 13, 2026

View reviewed changes

heyitsgrace996 added 3 commits April 13, 2026 14:28

Merge branch 'main' into grace.williams/fix-openai-async-streaming-span

86b9f68

Merge branch 'main' into grace.williams/fix-openai-async-streaming-span

5b97c9f

Merge branch 'main' into grace.williams/fix-openai-async-streaming-span

8f7c73e

chatgpt-codex-connector Bot reviewed Apr 13, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/openai/patch.py Outdated

chatgpt-codex-connector Bot reviewed Apr 13, 2026

View reviewed changes

Comment thread ddtrace/contrib/internal/openai/patch.py Outdated

heyitsgrace996 added 3 commits April 13, 2026 13:52

fix(openai): catch asyncio.CancelledError in async pre-parse guard [M…

9617b2e

…LOB-6778]

Merge branch 'main' into grace.williams/fix-openai-async-streaming-span

4aec9f0

Merge branch 'main' into grace.williams/fix-openai-async-streaming-span

e09925a

gh-worker-dd-mergequeue-cf854d Bot merged commit 2dda614 into main Apr 13, 2026
365 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the grace.williams/fix-openai-async-streaming-span branch April 13, 2026 19:17

heyitsgrace996 mentioned this pull request Apr 13, 2026

openai: async streaming spans never finished — AsyncAPIResponse.parse() unawaited in sync hook #17060

Closed

Navigation Menu

Uh oh!

fix(llmobs): pre-parse OpenAI AsyncAPIResponse to fix async streaming span lifecycle [MLOB-6778] - #17432

fix(llmobs): pre-parse OpenAI AsyncAPIResponse to fix async streaming span lifecycle [MLOB-6778]#17432
gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits into
mainfrom
grace.williams/fix-openai-async-streaming-span

heyitsgrace996 commented Apr 9, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 9, 2026 •

edited

Loading

Uh oh!

heyitsgrace996 commented Apr 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Apr 9, 2026

Uh oh!

emmettbutler left a comment

Uh oh!

ncybul left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

heyitsgrace996 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Root cause

Fix

Additional changes (beyond cherry-pick)

Before / After

Testing

Risks

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

heyitsgrace996 commented Apr 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Apr 9, 2026

Uh oh!

emmettbutler left a comment

Choose a reason for hiding this comment

Uh oh!

ncybul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

heyitsgrace996 commented Apr 9, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Apr 9, 2026 •

edited

Loading