Skip to content

Add namespace/job_name labels to opentelemetry_allocator_targets_remaining metric #4637

@hkf57

Description

@hkf57

Component(s)

target allocator

Is your feature request related to a problem? Please describe.

Problem

The opentelemetry_allocator_targets_remaining metric currently reports a scalar count of unallocated targets, making it impossible to distinguish
between:

  1. Intentionally filtered targets (e.g., namespaces excluded via allowNamespaces)
  2. Orphaned targets due to bugs (e.g., stale collector UIDs, version skew during rollouts)

This makes alerting extremely difficult - we can't set meaningful thresholds without either:

  • Creating false positives (alerting on intentional filters)
  • Missing real issues (high threshold that ignores single critical target losses)

Use Case

In our production environment:

  • We filter istio-system namespace targets (~1600 targets) to control costs
  • This baseline makes targets_remaining unusable for detecting orphaned targets
  • A recent incident (target assigned to deleted collector pod) went undetected for 14 hours

Describe the solution you'd like

Proposed Solution

Add labels to opentelemetry_allocator_targets_remaining to break down unallocated targets:

targetsRemaining = promauto.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: "opentelemetry_allocator_targets_remaining",
        Help: "Number of targets that could not be allocated to any collector",
    },
    []string{"job_name", "namespace"},  // Add these labels
)

This would allow alerts like:

  # Alert on orphaned targets, excluding known filtered namespaces
  opentelemetry_allocator_targets_remaining{namespace!="istio-system"} > 10

Alternative

If per-target labeling is too expensive (high cardinality), consider:

  • Add just reason label: {reason="namespace_filtered"} vs {reason="no_collectors"}
  • Or expose separate metrics: ..._targets_filtered vs ..._targets_orphaned

Describe alternatives you've considered

No response

Additional context

No response

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions