Summary
Collection nodes continuously report GossipSub metrics for cluster topics from past epochs (e.g., sync-cluster-cluster-* and consensus-cluster-*), even after those clusters are no longer active. This causes unbounded growth in metric cardinality over time.
Each epoch adds approximately 60 new metric series per collection node (2 topic prefixes × 3 clusters × ~10 metrics), and stale series from previous epochs are never cleaned up. This was confirmed by curling the metrics endpoint on collection-001, which showed active reporting of metrics for old cluster topics.
Impact
The top offenders by series count in Grafana are all histogram bucket metrics in the network_gossip_* family:
| Metric |
Series Count |
network_gossip_gossipsub_received_ihave_message_ids_bucket |
65.6K |
network_gossip_inbound_message_size_bytes_bucket |
61.9K |
network_gossip_outbound_message_size_bytes_bucket |
61.6K |
The top 3 metrics alone account for ~190K series (~28% of all flow_app series). While we are currently under our Grafana spend limit, the consistent upward trend will eventually become a cost concern.
Expected Behavior
When a collection node transitions to a new epoch and joins a new cluster, metrics for the previous epoch's cluster topics should stop being reported. Stale metric series should be cleaned up (e.g., by unregistering or resetting the associated Prometheus collectors).
This issue was created in collaboration with Claude.
Summary
Collection nodes continuously report GossipSub metrics for cluster topics from past epochs (e.g.,
sync-cluster-cluster-*andconsensus-cluster-*), even after those clusters are no longer active. This causes unbounded growth in metric cardinality over time.Each epoch adds approximately 60 new metric series per collection node (2 topic prefixes × 3 clusters × ~10 metrics), and stale series from previous epochs are never cleaned up. This was confirmed by curling the metrics endpoint on
collection-001, which showed active reporting of metrics for old cluster topics.Impact
The top offenders by series count in Grafana are all histogram bucket metrics in the
network_gossip_*family:network_gossip_gossipsub_received_ihave_message_ids_bucketnetwork_gossip_inbound_message_size_bytes_bucketnetwork_gossip_outbound_message_size_bytes_bucketThe top 3 metrics alone account for ~190K series (~28% of all
flow_appseries). While we are currently under our Grafana spend limit, the consistent upward trend will eventually become a cost concern.Expected Behavior
When a collection node transitions to a new epoch and joins a new cluster, metrics for the previous epoch's cluster topics should stop being reported. Stale metric series should be cleaned up (e.g., by unregistering or resetting the associated Prometheus collectors).
This issue was created in collaboration with Claude.