Add end-to-end test for app-extended-heartbeat telemetry event by khanayan123 · Pull Request #6338 · DataDog/system-tests

khanayan123 · 2026-02-18T15:40:31Z

Summary

Add a new TELEMETRY_EXTENDED_HEARTBEAT end-to-end scenario that validates the app-extended-heartbeat telemetry event across all SDK languages (Go, Java, .NET, C++, PHP, Ruby, Rust, Python, Node.js).

Changes

New scenario & test

utils/_context/_scenarios/__init__.py: New TELEMETRY_EXTENDED_HEARTBEAT scenario with shortened intervals (DD_TELEMETRY_HEARTBEAT_INTERVAL=1, DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL=2) so the event fires within test timeouts
tests/test_telemetry.py: New Test_ExtendedHeartbeat class — asserts that every config reported in app-started or app-client-configuration-change is eventually included in at least one app-extended-heartbeat event
utils/_features.py: Wired to existing feature flag app_extended_heartbeat_event (feature 77)

Manifest updates

All language manifests: Added version gates for Test_ExtendedHeartbeat at the appropriate tracer versions

Telemetry test fixes (drive-by)

Updated telemetry config name lookups to support languages that report both canonical (DD_*) and legacy config names
Added Ruby to enhanced config reporting precedence tests
Fixed type guards in float comparison for heartbeat interval validation

Test plan

CI passes for the TELEMETRY_EXTENDED_HEARTBEAT scenario across enabled languages
Existing telemetry tests continue to pass with updated config name lookups
Test_ExtendedHeartbeat correctly detects missing configs when app-extended-heartbeat events are present

🤖 Generated with Claude Code

Add comprehensive parametric test suite for app-extended-heartbeat telemetry event across all SDK languages. Tests verify timing, payload structure, consistency, and compliance with API spec. Tests added: - test_extended_heartbeat_emission: Verifies interval timing - test_extended_heartbeat_sequence: Validates multiple events - test_extended_heartbeat_payload_content: Checks required fields - test_extended_heartbeat_matches_app_started: Ensures consistency - test_extended_heartbeat_excludes_products_and_install_signature - test_extended_heartbeat_default_interval: Validates 24h default All tests run parametrically across 9 SDK languages (Go, Java, .NET, C++, PHP, Ruby, Rust, Python, Node.js). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

github-actions · 2026-02-18T15:41:06Z

CODEOWNERS have been resolved as:

manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/test_telemetry.py                                                 @DataDog/libdatadog-telemetry @DataDog/apm-sdk-capabilities @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

Register comprehensive parametric tests for app-extended-heartbeat event validation across all SDK languages. Tests verify: - Periodic emission timing (24h default, configurable) - Payload structure (config, deps, integrations) - Consistency with app-started event - Proper field exclusions (products, install_signature) Language version markers: - Go: v1.73.0-dev - Java: v1.40.0 - Node.js: v5.0.0 - Python: v2.0.0 - Ruby: v2.1.0-dev - C++: v0.2.0-dev - PHP: v1.0.0-dev - .NET: missing_feature (intentionally skipped - no forked runtime-id issues) Tests use fast intervals (0.3-0.5s) for CI efficiency. Test location: tests/parametric/test_telemetry.py::Test_ExtendedHeartbeat (lines 1256-1514) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Changed Test_ExtendedHeartbeat manifest entries from development version strings (e.g., v1.73.0-dev, v2.1.0-dev) to missing_feature to follow repository conventions. Development versions should not be specified in manifests. Tests for unreleased features should use missing_feature until the feature is actually released in a production version. Changes: - cpp.yml: v0.2.0-dev → missing_feature - golang.yml: v1.73.0-dev → missing_feature - nodejs.yml: '>=5.0.0' → missing_feature - php.yml: v1.0.0-dev → missing_feature - ruby.yml: v2.1.0-dev → missing_feature 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

…tionality Replace 6 complex test methods with a single simplified test that focuses on the core requirement: comparing configurations across telemetry events. The new test: - Grabs app-started, app-extended-heartbeat, and app-client-configuration-change events - Extracts configuration data from each - Asserts configs match between app-started and app-extended-heartbeat - Optionally validates config-change event configs match as well This reduces test complexity from ~260 lines to ~40 lines while maintaining coverage of the critical functionality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Configure manifests to run app-extended-heartbeat tests only on Node.js for initial validation: - Enable Node.js: *ref_5_0_0 (>=5.0.0) - Disable Python: v2.0.0 -> missing_feature - Disable Java: v1.40.0 -> missing_feature - Keep all others disabled: missing_feature (cpp, dotnet, php, golang, ruby) This allows focused testing of Node.js implementation before enabling other languages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

- Add SLF001 exception for test_telemetry.py to allow _get_telemetry_event usage - Fix nodejs.yml manifest to use '>=5.0.0' instead of non-existent anchor - All linting checks now pass (mypy, ruff, yamlfmt, yamllint, shellcheck) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Java tracer supports configurable extended heartbeat interval via DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL, so enable the parametric tests.

Resolve merge conflicts in manifests/java.yml, manifests/ruby.yml, and tests/parametric/test_telemetry.py. Keep Test_ExtendedHeartbeat additions while incorporating main's telemetry key normalization and stable configuration origin updates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Revert pyproject.toml changes (not needed for this PR) and remove DD_TELEMETRY_HEARTBEAT_INTERVAL from Test_ExtendedHeartbeat since we only need to configure the extended heartbeat interval. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Use integer interval (1s) for Java compatibility (getLong) - Assert configs as superset with value matching, order-agnostic - Build expected config from app-started + config-change overlay - Collect all config-change events, not just the last one Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…eat payload (#17203) ## Summary Fix the extended heartbeat payload key from `"configurations"` (plural) to `"configuration"` (singular) to match the [telemetry v2 API spec](https://github.com/DataDog/instrumentation-telemetry-api-docs/blob/main/GeneratedDocumentation/ApiDocs/v2/SchemaDocumentation/Schemas/app_extended_heartbeat.md) and align with other SDKs (Java, .NET, Node.js). ## Changes - **`ddtrace/internal/telemetry/writer.py`**: `payload["configurations"]` → `payload["configuration"]` - **`tests/telemetry/test_telemetry.py`**: Updated test assertions to match ## Motivation Cross-SDK system tests validate that `app-extended-heartbeat` payloads use the same schema. The spec and all other SDKs use `"configuration"` (singular). This mismatch would cause system test & dropped telemetry payloads failures for Python. ## Related - System test PR: DataDog/system-tests#6338 - Original Python implementation: #16628 Co-authored-by: ayan.khan <ayan.khan@datadoghq.com>

Resolve merge conflict in manifests/ruby.yml (kept both Test_ExtendedHeartbeat and updated Test_Stable_Configuration_Origin). Fix ruff lint issues: D209 docstring closing quotes and SLF001 private member access noqa annotations in test_telemetry.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

cbeauchesne

There is already test on heartbeats in DEFAULT scenario.

If you plan to test only one or two set of parameters, then the end-to-end DEFAULT scenario is a best fit from a quality POV as the coverage is by far more complete.

We can chat more about what are the good/bad use cases for parametric/end-to-end if you want, just ping me !

…cheduler (#1824) ## Summary Wire up the `DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL` config value to the telemetry scheduler. The env var was already parsed into `config.telemetry_extended_heartbeat_interval` but the scheduler hardcoded `Duration::from_secs(60 * 60 * 24)` instead of using it. Default remains 24h this only enables system tests to use a shorter interval to validate the `app-extended-heartbeat` event fires correctly. ## Changes - **`libdd-telemetry/src/worker/mod.rs`**: Replace hardcoded `60 * 60 * 24` with `config.telemetry_extended_heartbeat_interval` ## Motivation Cross-SDK system tests need to set a short extended heartbeat interval (e.g., 2s) to validate parity of the `app-extended-heartbeat` telemetry event across all SDKs. Without this fix, PHP and other libdatadog consumers cannot be system-tested for this feature. ## Related - System test PR: DataDog/system-tests#6338 Co-authored-by: edmund.kump <edmund.kump@datadoghq.com>

Add Test_ExtendedHeartbeat to tests/test_telemetry.py for the DEFAULT scenario, validating that extended heartbeat config is a superset of app-started plus any config changes. Set DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL env var in weblog containers. Add manifest entries for all libraries. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL from base container env to the DEFAULT scenario weblog_env where it belongs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The extended heartbeat validation is better suited as an end-to-end test in the DEFAULT scenario where it exercises the full tracer stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Relax the superset assertion to only validate that configs present in both app-started and app-extended-heartbeat have matching values. Some tracers may not track all configs in _sent_configs, so a strict superset check causes false failures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ogic" This reverts commit 1bfb4b8.

Use the last app-extended-heartbeat that appears after the last app-client-configuration-change in the telemetry stream. This ensures the heartbeat has had a chance to include all lazily registered configs (e.g. DD_LLMOBS_EVALUATOR_SAMPLING_RULES in Python which is registered during LLMObs.enable()). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The TELEMETRY_EXTENDED_HEARTBEAT scenario runs only this test, so the weblog is up for just a few seconds. Add a setup method that triggers a weblog request and waits 10s to ensure all lazy configs are registered and an extended heartbeat fires after the last config-change event. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Fix pytest setup method naming (setup_<name> matches test_<name>). Use interfaces.library.wait_for() to wait for the first extended heartbeat event, then sleep 5s to allow lazily registered configs to flush and a subsequent extended heartbeat to include them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The wait_for callback iterates all interface data, including entries without request.content. Use safe dict access to avoid AttributeError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… support (#4621) ## Summary Read `DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL` from the environment to allow configuring the extended heartbeat interval. This is only added so system tests can use a short interval to validate `app-extended-heartbeat` parity across all SDK tracers. Default remains 24h. ## Changes - **`internal/telemetry/client_config.go`**: Read `DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL` via `FloatEnv`, following the same pattern as `DD_TELEMETRY_HEARTBEAT_INTERVAL` ## Related - System test PR: DataDog/system-tests#6338 - Telemetry v2 API spec: https://github.com/DataDog/instrumentation-telemetry-api-docs/blob/main/GeneratedDocumentation/ApiDocs/v2/SchemaDocumentation/Schemas/app_extended_heartbeat.md Co-authored-by: ayan.khan <ayan.khan@datadoghq.com>

wait_for iterates all interface data including non-telemetry entries that lack request.content, causing AttributeError. Use time.sleep(15) which is the established pattern for telemetry setup methods. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use the SDK-provided tracer_time to determine which config events the extended heartbeat could have seen. Only compare against app-started and config-change events with tracer_time <= the extended heartbeat's tracer_time. This avoids false failures from lazily registered configs that arrive after the heartbeat. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Some weblogs (e.g. gunicorn with Python) fork workers that each have their own telemetry writer with a different runtime_id and different _sent_configs state. Filter all events by the runtime_id from app-started to ensure we compare events from the same process. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Instead of filtering by runtime_id, collect configs from ALL extended heartbeats and verify that every config from app-started/config-change was reported by at least one extended heartbeat with a tracer_time >= when the config was first reported. This handles forked workers (e.g. gunicorn) where different processes have different runtime_ids and different _sent_configs state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Drop tracer_time ordering checks — configs can appear in heartbeats before they are re-reported in config-change events (e.g. remote config updates). Simply verify every config name from app-started and config-change events appears in at least one extended heartbeat. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Follow the pattern of other telemetry setup methods which just trigger a weblog request without sleeping. The test checks across all collected heartbeats so no explicit wait is needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Set golang manifest to >2.7.1 (latest is v2.7.1, next release will include extended heartbeat support). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

dd-trace-go's extended heartbeat only includes configs from app-started, not from app-client-configuration-change events. The heartbeatEnricher.Transform() only handles AppStarted, AppDependenciesLoaded, and AppIntegrationChange payloads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…entire config state (#4633) ### What does this PR do? Replace the push-based config accumulation in `heartbeatEnricher` with a pull model that reads the full config state on demand from the `configuration` data source. **Problem:** `heartbeatEnricher` had an `AppStarted` case to capture startup configs, but it was unreachable because `appStartedReducer` consumed the `AppStarted` payload before `heartbeatEnricher` could see it. The extended heartbeat was missing all startup configurations. **Solution:** The `configuration` data source now retains all registered configs (using a `pending` set to track deltas for `Payload()`) and exposes an `All()` method. `heartbeatEnricher` calls `getConfigs()` when emitting the extended heartbeat to get the full current state directly from the source of truth. **Changes:** - `configuration.go` — Single map with `pending` set. `Add()` writes to config and marks pending. `Payload()` extracts only pending keys. New `All()` returns full accumulated state. - `internal/mapper/default.go` — `NewDefaultMapper` accepts `func() []transport.ConfKeyValue`. `heartbeatEnricher` pulls configs on demand instead of accumulating from pipeline. Removed `AppStarted` case. - `client.go` — Passes `configuration.All` to `NewDefaultMapper`. - `client_test.go` — Added integration tests for extended heartbeat config inclusion (single, multiple, dedup). ### Motivation Discovered while adding `app-extended-heartbeat` system tests in [DataDog/system-tests#6338](DataDog/system-tests#6338). The extended heartbeat was missing startup configs because `appStartedReducer` consumed them before `heartbeatEnricher` could see them. Rather than work around the pipeline ordering issue, this PR fixes the root cause by having `heartbeatEnricher` read config state directly from the source of truth. API spec for reference: https://github.com/DataDog/instrumentation-telemetry-api-docs/blob/main/GeneratedDocumentation/ApiDocs/v2/SchemaDocumentation/Schemas/app_extended_heartbeat.md Co-authored-by: ayan.khan <ayan.khan@datadoghq.com>

dd-trace-js merged support for DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL. Set manifest to >5.96.0 (latest is v5.96.0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

PHP telemetry runs via a sidecar process which does not emit app-extended-heartbeat events yet, even though libdatadog supports the DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL env var. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

khanayan123 and others added 6 commits February 18, 2026 14:45

remove md file

10c756f

This comment has been minimized.

Sign in to view

test(manifests): enable Test_ExtendedHeartbeat for Java

2cf6d0c

Java tracer supports configurable extended heartbeat interval via DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL, so enable the parametric tests.

khanayan123 mentioned this pull request Feb 26, 2026

chore(telemetry): add configurations to extended heartbeats DataDog/dd-trace-py#16628

Merged

khanayan123 and others added 3 commits March 30, 2026 22:12

khanayan123 mentioned this pull request Mar 31, 2026

feat(telemetry): add DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL config DataDog/dd-trace-js#7889

Merged

update tests and manifest file

2afc8ef

khanayan123 and others added 2 commits March 31, 2026 15:23

enable python system tests

f13a0a7

cbeauchesne reviewed Apr 1, 2026

View reviewed changes

khanayan123 and others added 4 commits April 1, 2026 09:56

Move extended heartbeat env var to DEFAULT scenario definition

1b11a18

Move DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL from base container env to the DEFAULT scenario weblog_env where it belongs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove parametric Test_ExtendedHeartbeat in favor of e2e test

58f28a8

The extended heartbeat validation is better suited as an end-to-end test in the DEFAULT scenario where it exercises the full tracer stack. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge branch 'main' into ayan.khan/app-extended-heartbeat

e5dfac1

khanayan123 and others added 4 commits April 1, 2026 14:26

enable ruby system tests

4dfaa04

Revert "fix(telemetry): improve extended heartbeat config assertion l…

63b9cf4

…ogic" This reverts commit 1bfb4b8.

khanayan123 force-pushed the ayan.khan/app-extended-heartbeat branch from b0c8500 to 388375a Compare April 1, 2026 18:57

khanayan123 and others added 3 commits April 2, 2026 08:35

Fix wait_for lambda to handle non-telemetry data safely

03bca5b

The wait_for callback iterates all interface data, including entries without request.content. Use safe dict access to avoid AttributeError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

khanayan123 and others added 8 commits April 2, 2026 09:41

Enable extended heartbeat test for next Go release

11e4845

Set golang manifest to >2.7.1 (latest is v2.7.1, next release will include extended heartbeat support). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

khanayan123 mentioned this pull request Apr 2, 2026

fix(telemetry): app-extended-heartbeat event not correctly reporting entire config state DataDog/dd-trace-go#4633

Merged

pass golang

0ed912f

khanayan123 and others added 2 commits April 9, 2026 11:12

Merge branch 'main' into ayan.khan/app-extended-heartbeat

4b9c4ec

Enable extended heartbeat test for next Node.js release

9d3aa89

dd-trace-js merged support for DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL. Set manifest to >5.96.0 (latest is v5.96.0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

khanayan123 force-pushed the ayan.khan/app-extended-heartbeat branch from 6980c96 to 9d3aa89 Compare April 9, 2026 16:41

enable php

e9f257e

khanayan123 changed the title ~~feat(telemetry): add app-extended-heartbeat parametric tests~~ Add end-to-end test for app-extended-heartbeat telemetry event Apr 14, 2026

khanayan123 changed the title ~~Add end-to-end test for app-extended-heartbeat telemetry event~~ Add end-to-end weblog test for app-extended-heartbeat telemetry event Apr 14, 2026

khanayan123 changed the title ~~Add end-to-end weblog test for app-extended-heartbeat telemetry event~~ Add end-to-end test for app-extended-heartbeat telemetry event Apr 14, 2026

khanayan123 and others added 3 commits April 14, 2026 10:32

Merge branch 'main' into ayan.khan/app-extended-heartbeat

0742b54

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix version for golang

9477f2c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add end-to-end test for app-extended-heartbeat telemetry event#6338

Add end-to-end test for app-extended-heartbeat telemetry event#6338
khanayan123 wants to merge 46 commits intomainfrom
ayan.khan/app-extended-heartbeat

khanayan123 commented Feb 18, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 18, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

cbeauchesne left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

khanayan123 commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

New scenario & test

Manifest updates

Telemetry test fixes (drive-by)

Test plan

Uh oh!

github-actions bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

khanayan123 commented Feb 18, 2026 •

edited

Loading

github-actions bot commented Feb 18, 2026 •

edited

Loading