Skip to content

perf(telemetry): skip non-recording LLM trace work#6371

Draft
dexhunter wants to merge 1 commit into
google:mainfrom
dexhunter:perf/skip-non-recording-llm-trace
Draft

perf(telemetry): skip non-recording LLM trace work#6371
dexhunter wants to merge 1 commit into
google:mainfrom
dexhunter:perf/skip-non-recording-llm-trace

Conversation

@dexhunter

Copy link
Copy Markdown

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:

trace_call_llm builds telemetry configuration and serializes request/response content before writing span attributes, even when the supplied or current span is not recording. Those values cannot be retained by a non-recording span, so large LLM payloads incur avoidable CPU work.

Solution:

Resolve the span first and return immediately when span.is_recording() is false. Recording spans keep the existing telemetry and attribute path unchanged. The new unit test verifies that a non-recording span does not read telemetry configuration, serialize content, or write attributes.

This addresses one narrow source of the repeated serialization reported in #4233; it does not claim to eliminate every serialization path in that issue.

Performance

An instrumented helper-level benchmark used content capture with a synthetic 108,167-byte response:

Current main This PR
trace_call_llm, non-recording span 139.820 us 0.123 us

This avoids about 99.91% of the helper overhead in that case (approximately 1,137x). It is not an end-to-end agent or model latency claim. Recording-span behavior is covered by correctness gates and remains unchanged.

The early-guard direction was identified through autoresearch with Weco. The final patch was rebuilt on current main and hardened with recording and non-recording counterexamples.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Results:

  • pytest tests/unittests/telemetry/test_spans.py -q: 80 passed
  • unit-test tox environments Python 3.10 through 3.14: all passed
  • strict evaluator: recording-span contracts passed; non-recording result 0.123 us
  • ruff check, pyink --check, and git diff --check: passed

The repository-wide tox run was also attempted without exclusions. Its only collection blocker was the checkout-external open_source_workspace/.agents/skills/adk-verify-snippets/scripts/verify_md.py, which is not present in this checkout. Re-running all unit tests while excluding only test_verify_snippets.py passed on Python 3.10 through 3.14.

Manual End-to-End (E2E) Tests:

Not applicable for this pure tracing guard. The unit test and strict evaluator exercise both an explicitly supplied span and the normal recording/non-recording behavior without requiring an external model request.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end. (Not applicable; see above.)
  • Any dependent changes have been merged and published in downstream modules. (No dependencies.)

Additional context

The commit is cryptographically signed and verified by GitHub.

Avoid telemetry configuration and payload serialization when the active span cannot record attributes.

Refs google#4233
@adk-bot adk-bot added the tracing [Component] This issue is related to OpenTelemetry tracing label Jul 10, 2026
@dexhunter dexhunter marked this pull request as draft July 11, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tracing [Component] This issue is related to OpenTelemetry tracing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants