Add Gardener Receiver E2E Test by chrkl · Pull Request #67 · gardener/opentelemetry-collector

chrkl · 2026-07-29T10:29:53Z

How to categorize this PR?

/kind test

What this PR does / why we need it:

Add e2e test for the Gardener metrics receiver

Which issue(s) this PR fixes:
gardener/observability#57

Special notes for your reviewer:

Release note:

Add Gardener Receiver e2e test

gardener-prow · 2026-07-29T10:29:59Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rickardsjp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hyperspace-pr-bot · 2026-07-29T10:30:14Z

Summary

The following content is AI-generated and provides a summary of the pull request:

/kind test

What this PR does / why we need it:

Adds an end-to-end integration test for the Gardener metrics receiver. The test spins up a KinD cluster with a full Gardener deployment, then verifies that the OpenTelemetry Collector correctly scrapes and exports Gardener resource metrics (shoots, seeds, projects, managed seeds, gardenlets) via both the Prometheus exporter and the OTLP HTTP exporter.

Key additions:

.github/workflows/gardener-integration.yaml — New GitHub Actions workflow that triggers on PRs (opened/edited/reopened), on a nightly schedule (0 4 * * *), or manually via workflow_dispatch (with an optional gardener_ref input). It checks out both this repo and gardener/gardener, sets up Go, and runs the integration test script.
hack/gardener-integration-test.sh — Shell script that:
1. Brings up a KinD cluster using Gardener's make kind-up
2. Deploys Gardener via make gardener-up
3. Resolves the latest snapshot collector image from the container registry
4. Deploys two OpenTelemetryCollector instances — one with a Prometheus exporter, one with an OTLP HTTP exporter
5. Deploys a Prometheus Operator instance with a ServiceMonitor and OTLP write receiver enabled
6. Creates a sample Shoot resource
7. Polls Prometheus to verify both garden_shoot_info (scraped path) and garden.shoot.info (OTLP push path) metrics are present within a 5-minute timeout
8. Tears down the KinD cluster on exit (skippable via SKIP_TEARDOWN)

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Add Gardener integration test for the Gardener metrics receiver, covering both Prometheus scrape and OTLP push export paths.

🔄 Regenerate and Update Summary
✏️ Insert as PR Description (deletes this comment)
🗑️ Delete comment

PR Bot Information

Version: 1.29.6

Event Trigger: pull_request.opened
Summary Prompt: Default Prompt
Correlation ID: 706e9200-8b38-11f1-833e-5f9a5d86dcbe
LLM: anthropic--claude-4.6-sonnet
Output Template: Repository PR Template
File Content Strategy: Full file content

hyperspace-pr-bot

The PR introduces a Gardener integration test workflow and script with several substantive issues: the most critical is that resolve_collector_image will always fail because the Google Artifact Registry tag-list API does not include the Docker Hub-specific .manifest field, meaning the test can never successfully run. Additionally, the missing synchronize trigger means new commits pushed to a PR won't trigger the test, two endgroup calls are missing on early-return error paths producing malformed CI log output, and actions/setup-go@v6 diverges from the @v7 version used everywhere else in the repository.

PR Bot Information

Version: 1.29.6

Event Trigger: pull_request.opened
Correlation ID: 706e9200-8b38-11f1-833e-5f9a5d86dcbe
LLM: anthropic--claude-4.6-sonnet
Experiment Variant: control
File Content Strategy: Full file content

chrkl · 2026-07-29T10:31:30Z

/hold

chrkl · 2026-07-29T12:44:54Z

/unhold

chrkl · 2026-08-07T12:24:00Z

/hold

vicwicker · 2026-08-13T13:31:33Z

+	return &monitoringv1.ServiceMonitor{
+		TypeMeta: metav1.TypeMeta{APIVersion: monitoringv1.SchemeGroupVersion.String(), Kind: monitoringv1.ServiceMonitorsKind},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "gardener-prometheus-collector",


Maybe we could use the constant defined above.

vicwicker · 2026-08-13T13:31:54Z

+
+// serviceMonitorLabel is the label the Prometheus instance selects
+// ServiceMonitors by, and that the ServiceMonitor carries.
+var serviceMonitorLabel = map[string]string{"app": "gardener-prometheus-collector"}


Maybe we could use the constant defined above.

vicwicker · 2026-08-13T13:32:13Z

+		},
+		Spec: monitoringv1.ServiceMonitorSpec{
+			Selector: metav1.LabelSelector{MatchLabels: map[string]string{
+				"app.kubernetes.io/name":                           "gardener-prometheus-collector",


Maybe we could use the constant defined above.

vicwicker · 2026-08-13T13:35:22Z

+func prometheusCollector(namespace, image string) *otelv1beta1.OpenTelemetryCollector {
+	return &otelv1beta1.OpenTelemetryCollector{
+		TypeMeta:   metav1.TypeMeta{APIVersion: otelv1beta1.GroupVersion.String(), Kind: otelCollectorKind},
+		ObjectMeta: metav1.ObjectMeta{Name: "gardener-prometheus", Namespace: namespace},


I thought maybe we could extract "gardener-prometheus" into its own constant next to prometheusCollectorDeployment to show one is the collector and the other the deployment that the OTel operator deploys.

vicwicker · 2026-08-13T13:35:59Z

+)
+
+// deployComponents replicates the bulk of the former shell
+// run_collector_e2e_test: it creates the test namespace, the Gardener API


deployComponents replicates the bulk of the former shell run_collector_e2e_test

Does this sentence apply? Was there such a script before, or only in your PR?

vicwicker · 2026-08-13T13:38:05Z

+		return 0, fmt.Errorf("parse sample value %q: %w", raw, err)
+	}
+	return value, nil
+}


Maybe it would be a smaller improvement if we used a Prometheus client instead, and worked with the objects it returns.

vicwicker · 2026-08-13T13:39:16Z

+        }
+    ]
+    | max_by(.uploaded).tag // empty
+  ')


Is there an alternative way to find the collector image from the repository directly? Like, do we have some pinned dependency somewhere that we could use to reference to find the image?

The test script attempts to modify system files and run service commands without sudo, assuming it runs as root. Use `sudo -E` to preserve the environment and pass PATH explicitly to maintain access to the Go/KinD/kubectl toolchain.

Gardener's CI setup requires dnsmasq but ubuntu-latest runners don't include it. Disable the systemd unit to prevent it from binding port 53 before Gardener's script configures it.

This disables some Prow specific features in Gardener like pushing metrics to Prometheus remote write.

This was added for testing purposes only.

Replace direct script invocation with `make test-e2e` in CI workflow and add corresponding make target.

Move metric validation from shell script polling into a build-tagged Go test suite using Ginkgo/Gomega. This allows the validation to run independently against an already-provisioned cluster via `make test-e2e-validate`. The test establishes a port-forward to Prometheus and asserts on both the scraped garden_shoot_info metric and the OTLP-pushed garden.shoot.info metric.

The e2e suite now deploys the full stack (collectors, Prometheus, Shoot) in addition to validating metrics. Previously deployment was handled entirely by the shell script; this migration to Go gives us better testing semantics and easier debugging. Add dependencies on the otel-operator and prometheus-operator API packages to support programmatic resource creation. Refactor the shell script to focus on KinD/Gardener provisioning while the Go suite handles component deployment.

Install ginkgo/v2 and its dependencies, then migrate the e2e test command from `go test` to the ginkgo CLI for better test output formatting and GitHub Actions integration.

Replace the port-forward mechanism with a LoadBalancer service that exposes Prometheus directly. This simplifies the test setup by relying on Gardener's provider-local cloud-controller to assign an external IP from the configured range.

chrkl requested a review from a team as a code owner July 29, 2026 10:29

gardener-prow Bot added the kind/test Test label Jul 29, 2026

gardener-prow Bot added cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 29, 2026

hyperspace-pr-bot Bot reviewed Jul 29, 2026

View reviewed changes

chrkl mentioned this pull request Jul 29, 2026

☂️ Replace gardener-metrics-exporter with OpenTelemetry Collector gardener/observability#57

Open

7 tasks

gardener-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 29, 2026

chrkl force-pushed the gardener-integration branch 6 times, most recently from 09c8e23 to 0d4f655 Compare July 29, 2026 12:44

gardener-prow Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 29, 2026

chrkl force-pushed the gardener-integration branch from 0d4f655 to 2657451 Compare July 29, 2026 13:30

chrkl changed the title ~~Add Gardener integration test~~ Add Gardener Receiver E2E test Aug 7, 2026

chrkl changed the title ~~Add Gardener Receiver E2E test~~ Add Gardener Receiver E2E Test Aug 7, 2026

chrkl force-pushed the gardener-integration branch 2 times, most recently from f5e8420 to 7cb9047 Compare August 7, 2026 11:28

gardener-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 7, 2026

chrkl force-pushed the gardener-integration branch from f57026a to a227420 Compare August 7, 2026 12:53

gardener-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 7, 2026

chrkl force-pushed the gardener-integration branch 2 times, most recently from 30fdc33 to 98d4db6 Compare August 11, 2026 12:52

vicwicker reviewed Aug 13, 2026

View reviewed changes

chrkl added 14 commits August 14, 2026 10:03

Add Gardener receiver integration test script

80dabb2

Add test for OTLP based metric ingestion

717e99d

Add Gardener integration test workflow

758e575

Run gardener integration test as root

beac863

The test script attempts to modify system files and run service commands without sudo, assuming it runs as root. Use `sudo -E` to preserve the environment and pass PATH explicitly to maintain access to the Go/KinD/kubectl toolchain.

Install dnsmasq for Gardener CI tests

4c0941e

Gardener's CI setup requires dnsmasq but ubuntu-latest runners don't include it. Disable the systemd unit to prevent it from binding port 53 before Gardener's script configures it.

Disable CI environment variable

26009a3

This disables some Prow specific features in Gardener like pushing metrics to Prometheus remote write.

Remove pull_request trigger from Gardener workflow

8ed6924

This was added for testing purposes only.

Rename and move to gardenerreceiver directory

556593d

Add Apache 2.0 license header to gardener receiver e2e test script

0854f32

Use make target for e2e tests

8aa3af1

Replace direct script invocation with `make test-e2e` in CI workflow and add corresponding make target.

Add Ginkgo v2 testing framework to gardener receiver

20a8cbc

Install ginkgo/v2 and its dependencies, then migrate the e2e test command from `go test` to the ginkgo CLI for better test output formatting and GitHub Actions integration.

Replace port-forwarding with LoadBalancer service

63e4ca1

Replace the port-forward mechanism with a LoadBalancer service that exposes Prometheus directly. This simplifies the test setup by relying on Gardener's provider-local cloud-controller to assign an external IP from the configured range.

chrkl force-pushed the gardener-integration branch from 86bdc8a to 63e4ca1 Compare August 14, 2026 10:05

Uh oh!

Conversation

chrkl commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gardener-prow Bot commented Jul 29, 2026

Uh oh!

hyperspace-pr-bot Bot commented Jul 29, 2026

Summary

Uh oh!

hyperspace-pr-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrkl commented Jul 29, 2026

Uh oh!

chrkl commented Jul 29, 2026

Uh oh!

chrkl commented Aug 7, 2026

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vicwicker Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chrkl commented Jul 29, 2026 •

edited

Loading