[target-allocator] Add feature flag for labeled targets_remaining metric with OTel conventions by hkf57 · Pull Request #4640 · open-telemetry/opentelemetry-operator

hkf57 · 2026-01-20T21:29:23Z

Description

Adds k8s.namespace.name and job.name labels to the opentelemetry_allocator_targets_remaining metric (when enabled via feature gate) to enable distinguishing between intentionally filtered targets and orphaned targets due to bugs.

Breaking Change Mitigation: This change is behind feature gates (alpha) to provide a migration path for existing users.

Motivation and Context

Currently, opentelemetry_allocator_targets_remaining reports a scalar count, making it impossible to create meaningful alerts. In production environments with namespace filtering (e.g., excluding istio-system to control costs), the baseline can be ~1600 targets. This masks genuine issues like orphaned targets from deleted collectors or version skew scenarios.

Solution

Feature Gates (Alpha)

Operator gate: operator.targetallocator.labeledmetrics

Registered in pkg/featuregate/featuregate.go
When enabled, the operator sets --feature-gates=target-allocator.labeled-metrics on the TA container args

Target Allocator gate: target-allocator.labeled-metrics

Registered in cmd/otel-allocator/internal/featuregate/featuregate.go
TA parses --feature-gates via CLI flags
Controls labelled metric emission at runtime

This two-gate approach follows the reviewer's guidance: the operator gate propagates to the TA via --feature-gates, avoiding any config struct changes that would be a breaking change to remove when the gate graduates.

Label Names (OTel Semantic Conventions)

job.name
k8s.namespace.name

Implementation

Modified SetTargets() in allocator.go to:

Check feature gate state
When enabled: Group remaining targets by job and namespace, emit labelled metrics
When disabled: Emit single unlabelled gauge (original behaviour)
Extract namespace from __meta_kubernetes_namespace label
Use "unknown" for targets without namespace metadata

Usage

Enable via the operator's feature gates:

--feature-gates=operator.targetallocator.labeledmetrics

The operator propagates this to the target allocator container automatically.

After this change (when feature enabled)

opentelemetry_allocator_targets_remaining{k8s_namespace_name!="istio-system"} > 10
opentelemetry_allocator_targets_remaining{job_name="kube-state-metrics"} > 0

Testing

Code compiles successfully
Mock interfaces updated
Feature gates registered (operator + TA)
Container args propagation test
User-specified feature gates not overridden
Unit tests for labelled/unlabelled behaviour
E2E tests with feature flag enabled/disabled

Fixes #4637

linux-foundation-easycla · 2026-01-20T21:29:31Z

The committers listed above are authorized under a signed CLA.

✅ login: hkf57 / name: Charles Lee (356a181, 4c54bc7, 915c38b, bbaa28c, c142769, c6b0456)

swiatekm · 2026-01-30T18:41:22Z

I don't think what you've done is enough for the feature gate to propagate down. Right now, if you set it on the operator, it won't do anything. It is possible to set it on the allocator directly by passing it as an argument, but that's quite inconvenient. It'd be better if the operator did it if it's enabled.

swiatekm · 2026-04-08T10:35:17Z

I'd prefer if we made this a feature flag in the target allocator as well.

Add a featuregates.go in cmd/otel-allocator similar to the one in pkg/featuregates.go
Add the target allocator feature gates to command line flags in main.go
Add your new feature gate and use it for any checks
The operator feature gate stays, and causes the operator to set the feature gate on the target allocator

Not only will this remove the need to set a new attribute on the structs in target allocator, you also won't need to add a new config option. Adding that option and then removing it later will be an unnecessary breaking change.

swiatekm · 2026-04-09T13:19:38Z


-	a.targetsRemaining.Record(context.Background(), int64(len(targets)))
+	// Record remaining targets, with labels if feature flag is enabled
+	if a.useLabeledMetrics {


I think you can just use the feature gate directly here and avoid passing this option down from main.go.

github-actions · 2026-04-09T13:51:47Z

E2E Test Results

32 files 251 suites 2h 6m 52s ⏱️
97 tests 97 ✅ 0 💤 0 ❌
255 runs 255 ✅ 0 💤 0 ❌

Results for commit c6b0456.

♻️ This comment has been updated with latest results.

…ning metric Add namespace and job_name labels to opentelemetry_allocator_targets_remaining metric to allow distinguishing between intentionally filtered targets and orphaned targets due to bugs. Currently, the metric reports a scalar count making it impossible to alert on orphaned targets without false positives from intentionally filtered namespaces (e.g., istio-system excluded for cost control). With this change, alerts can filter specific namespaces: opentelemetry_allocator_targets_remaining{namespace!="istio-system"} > 10 Changes: - Modified SetTargets() to group remaining targets by job_name and namespace - Each unique job_name/namespace combination emits separate gauge value - Targets without namespace use "unknown" label value - All existing tests pass Fixes open-telemetry#4637 Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

…ric with OTel conventions - Add operator.targetallocator.labeledmetrics feature gate (alpha) - Change label names to OTel semantic conventions: job.name and k8s.namespace.name - Maintain backward compatibility when feature flag is disabled Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

Move feature gate propagation to follow the existing pattern: operator checks the gate and writes config into the TA configmap, rather than passing CLI flags. The TA reads labeled_metrics from its config file. Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

…nfig Add TA-local feature gate (target-allocator.labeled-metrics) and have the operator propagate it via --feature-gates on the container args. Avoids adding a config option that would be a breaking change to remove. Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

…base Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

github-actions · 2026-07-09T10:14:13Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2026-07-27T10:24:30Z

Closed as inactive. Feel free to reopen if this PR is still being worked on.

hkf57 requested a review from a team as a code owner January 20, 2026 21:29

hkf57 changed the title ~~[target-allocator] Add namespace and job_name labels to targets_remaining metric~~ [target-allocator] Add feature flag for labeled targets_remaining metric with OTel conventions Jan 28, 2026

hkf57 force-pushed the add-labels-to-targets-remaining branch 3 times, most recently from c163f8d to dd71375 Compare April 2, 2026 06:31

hkf57 force-pushed the add-labels-to-targets-remaining branch from df64187 to 057b77c Compare April 8, 2026 23:04

swiatekm reviewed Apr 9, 2026

View reviewed changes

hkf57 added 6 commits April 14, 2026 09:49

Add changelog entry for labeled targets_remaining metric

bbaa28c

Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

[target-allocator] Fix lint, formatting, and feature gate ID after re…

c6b0456

…base Signed-off-by: Charles Lee <charles.lee@safetyculture.io>

hkf57 force-pushed the add-labels-to-targets-remaining branch from 057b77c to c6b0456 Compare April 13, 2026 23:59

hkf57 requested a review from swiatekm April 15, 2026 03:51

github-actions Bot added the stale label Jul 9, 2026

github-actions Bot closed this Jul 27, 2026

Navigation Menu

Uh oh!

[target-allocator] Add feature flag for labeled targets_remaining metric with OTel conventions - #4640

[target-allocator] Add feature flag for labeled targets_remaining metric with OTel conventions#4640
hkf57 wants to merge 6 commits into
open-telemetry:mainfrom
hkf57:add-labels-to-targets-remaining

hkf57 commented Jan 20, 2026 •

edited

Loading

Uh oh!

linux-foundation-easycla Bot commented Jan 20, 2026 •

edited

Loading

Uh oh!

swiatekm commented Jan 30, 2026

Uh oh!

swiatekm commented Apr 8, 2026

Uh oh!

swiatekm Apr 9, 2026

Uh oh!

github-actions Bot commented Apr 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

hkf57 commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Solution

Feature Gates (Alpha)

Label Names (OTel Semantic Conventions)

Implementation

Usage

After this change (when feature enabled)

Testing

Uh oh!

linux-foundation-easycla Bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swiatekm commented Jan 30, 2026

Uh oh!

swiatekm commented Apr 8, 2026

Uh oh!

swiatekm Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hkf57 commented Jan 20, 2026 •

edited

Loading

linux-foundation-easycla Bot commented Jan 20, 2026 •

edited

Loading

github-actions Bot commented Apr 9, 2026 •

edited

Loading