Component(s)
target allocator
Is your feature request related to a problem? Please describe.
Problem
The opentelemetry_allocator_targets_remaining metric currently reports a scalar count of unallocated targets, making it impossible to distinguish
between:
- Intentionally filtered targets (e.g., namespaces excluded via
allowNamespaces)
- Orphaned targets due to bugs (e.g., stale collector UIDs, version skew during rollouts)
This makes alerting extremely difficult - we can't set meaningful thresholds without either:
- Creating false positives (alerting on intentional filters)
- Missing real issues (high threshold that ignores single critical target losses)
Use Case
In our production environment:
- We filter
istio-system namespace targets (~1600 targets) to control costs
- This baseline makes
targets_remaining unusable for detecting orphaned targets
- A recent incident (target assigned to deleted collector pod) went undetected for 14 hours
Describe the solution you'd like
Proposed Solution
Add labels to opentelemetry_allocator_targets_remaining to break down unallocated targets:
targetsRemaining = promauto.NewGaugeVec(
prometheus.GaugeOpts{
Name: "opentelemetry_allocator_targets_remaining",
Help: "Number of targets that could not be allocated to any collector",
},
[]string{"job_name", "namespace"}, // Add these labels
)
This would allow alerts like:
# Alert on orphaned targets, excluding known filtered namespaces
opentelemetry_allocator_targets_remaining{namespace!="istio-system"} > 10
Alternative
If per-target labeling is too expensive (high cardinality), consider:
- Add just reason label: {reason="namespace_filtered"} vs {reason="no_collectors"}
- Or expose separate metrics: ..._targets_filtered vs ..._targets_orphaned
Describe alternatives you've considered
No response
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Component(s)
target allocator
Is your feature request related to a problem? Please describe.
Problem
The
opentelemetry_allocator_targets_remainingmetric currently reports a scalar count of unallocated targets, making it impossible to distinguishbetween:
allowNamespaces)This makes alerting extremely difficult - we can't set meaningful thresholds without either:
Use Case
In our production environment:
istio-systemnamespace targets (~1600 targets) to control coststargets_remainingunusable for detecting orphaned targetsDescribe the solution you'd like
Proposed Solution
Add labels to
opentelemetry_allocator_targets_remainingto break down unallocated targets:This would allow alerts like:
Alternative
If per-target labeling is too expensive (high cardinality), consider:
Describe alternatives you've considered
No response
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding
+1orme too, to help us triage it. Learn more here.