Skip to content

OCPNODE-4055: Add e2e testcase for additional storage support feature#31399

Draft
BhargaviGudi wants to merge 1 commit into
openshift:mainfrom
BhargaviGudi:additionalImageStore
Draft

OCPNODE-4055: Add e2e testcase for additional storage support feature#31399
BhargaviGudi wants to merge 1 commit into
openshift:mainfrom
BhargaviGudi:additionalImageStore

Conversation

@BhargaviGudi

@BhargaviGudi BhargaviGudi commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

This PR adds E2E testcase for the AdditionalStorageConfig feature (TechPreviewNoUpgrade)
Test Coverage

E2E Test (additional_storage_e2e.go)

1 comprehensive test in openshift/disruptive-longrunning suite:

"should functionally verify all three storage types work together"

  • Creates single-node MachineConfigPool for faster rollouts (~10-15 min vs ~25 min)
  • Pre-populates multi-arch test image to additional image store using podman
  • Configures ContainerRuntimeConfig with additionalImageStores
  • Verifies storage.conf contains the configured image store path
  • Creates test pod that successfully accesses pre-populated image
  • Tests fallback behavior when image is removed from additional store
  • Validates all storage types (image/artifact/layer stores) work together

Implementation Details

Single-Node MCP Approach

  • Why: Faster rollouts for E2E tests (10-15 min vs 25+ min for full worker pool)
  • How: createSingleNodeMCP() creates custom MCP targeting single worker node
  • SNO Compatible: Works on Single Node OpenShift clusters

Testing:
Test case passed on local cluster

Summary by CodeRabbit

  • Tests
    • Added a disruptive end-to-end test for “Additional Storage,” validating additional image, layer, and artifact store behavior when the feature is enabled.
    • Confirms configuration propagation (including storage.conf and CRI-O settings), ensures CRI-O remains active, and performs artifact read/write validation.
    • Verifies image store usage with pre-populated images and validates registry pull fallback after removing an image from the additional store.
    • Adds improved helpers for node-pinned pods and single-node MachineConfigPool rollout/readiness checks.
  • Documentation
    • Updated the extended node test README to include the new Additional Storage E2E case.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot

openshift-ci-robot commented Jul 17, 2026

Copy link
Copy Markdown

@BhargaviGudi: This pull request references OCPNODE-4055 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

This PR adds E2E testcase for the AdditionalStorageConfig feature (TechPreviewNoUpgrade)
Test Coverage

E2E Test (additional_storage_e2e.go)

1 comprehensive test in openshift/disruptive-longrunning suite:

"should functionally verify all three storage types work together"

  • Creates single-node MachineConfigPool for faster rollouts (~10-15 min vs ~25 min)
  • Pre-populates multi-arch test image to additional image store using podman
  • Configures ContainerRuntimeConfig with additionalImageStores
  • Verifies storage.conf contains the configured image store path
  • Creates test pod that successfully accesses pre-populated image
  • Tests fallback behavior when image is removed from additional store
  • Validates all storage types (image/artifact/layer stores) work together

Implementation Details

Single-Node MCP Approach

  • Why: Faster rollouts for E2E tests (10-15 min vs 25+ min for full worker pool)
  • How: createSingleNodeMCP() creates custom MCP targeting single worker node
  • SNO Compatible: Works on Single Node OpenShift clusters

Testing:
Test case passed on local cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 17, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 17, 2026
@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: BhargaviGudi
Once this PR has been reviewed and has the lgtm label, please assign rphillips for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds reusable single-node MachineConfigPool lifecycle and polling helpers, plus a feature-gated E2E suite that configures and validates CRI-O additional artifact, image, and layer stores, including registry fallback behavior.

Changes

Additional storage validation

Layer / File(s) Summary
Single-node MCP lifecycle
test/extended/node/node_mcp_helpers.go, test/extended/node/node_utils.go
Adds custom-role detection, isolated MCP creation and cleanup, rollout handling, and MCP readiness polling.
Test orchestration utilities
test/extended/node/node_utils.go
Adds secured pod creation, runtime configuration polling, directory management, pod cleanup, and MCP transition helpers.
Additional storage end-to-end flow
test/extended/node/additional_storage_e2e.go, test/extended/node/README.md
Adds a feature-gated test covering storage setup, ContainerRuntimeConfig rollout, CRI-O configuration checks, image-store usage, registry fallback, and suite documentation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AdditionalStorageE2E
  participant MachineConfigOperator
  participant Node
  participant CRIO
  participant Registry
  AdditionalStorageE2E->>Node: Create storage directories and preload image
  AdditionalStorageE2E->>MachineConfigOperator: Create ContainerRuntimeConfig
  MachineConfigOperator->>Node: Roll out storage configuration
  AdditionalStorageE2E->>CRIO: Verify storage configuration and service status
  AdditionalStorageE2E->>Node: Start pod from additional image store
  AdditionalStorageE2E->>Node: Remove stored image and start fallback pod
  Node->>Registry: Pull image after additional store miss
Loading

Possibly related PRs


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error additional_storage_e2e.go logs raw pod logs and podObj.Status (%+v), which can leak internal IPs/IDs or any data emitted by the container. Log only specific sanitized fields (reason/message) and avoid dumping full pod status or raw logs; redact IPs/node names or gate behind debug.
Docstring Coverage ⚠️ Warning Docstring coverage is 29.41% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning Bare Expect calls remain, podman pull is still unbounded, and DeferCleanup removes directories before MCP rollback, so test quality is not met. Add a timeout-bounded image pull, add failure messages to the main assertions, and register cleanup so MCP/ContainerRuntimeConfig rollback runs before node directories are removed.
✅ Passed checks (12 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All new Ginkgo titles are static; no dynamic names, IDs, timestamps, or concatenated titles were found.
Microshift Test Compatibility ✅ Passed PASS: The new e2e test is already gated with [apigroup:config.openshift.io][apigroup:machineconfiguration.openshift.io], and the shared helper also skips MicroShift before MachineConfig use.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The new e2e test uses only one worker node and a custom MCP on that same node; it has no multi-node, failover, or cross-node scheduling assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed This PR only adds test helpers and an E2E test; I found no new deployment/controller scheduling constraints like anti-affinity, spread constraints, or replica logic.
Ote Binary Stdout Contract ✅ Passed No main/init/TestMain/BeforeSuite stdout writes were added; the new code only logs inside Ginkgo test/helpers, and NewCLI is stdout-free.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The test uses a public registry image, but it is tagged [Skipped:Disconnected], and I found no IPv4-only assumptions in the new test/helpers.
No-Weak-Crypto ✅ Passed No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons appear in the PR changes.
Container-Privileges ✅ Passed No privileged pod settings were added; createTestPod uses runAsNonRoot, runAsUser 1000, allowPrivilegeEscalation false, and drops ALL capabilities.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding an e2e test for the additional storage support feature.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (2)
test/extended/node/node_utils.go (1)

1174-1265: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Keep one MCP readiness implementation.

This duplicates the existing waitForMCP logic at Lines 556-618. Export or wrap the existing helper so readiness semantics cannot diverge.

As per coding guidelines, favor clarity and maintainability over cleverness.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/node/node_utils.go` around lines 1174 - 1265, Remove the
duplicated readiness logic from WaitForMCP and reuse the existing waitForMCP
implementation near the earlier MCP wait code by exporting or wrapping it.
Preserve the current options, timeout, draining behavior, degraded-pool
handling, and readiness semantics through that single implementation.

Source: Coding guidelines

test/extended/node/node_mcp_helpers.go (1)

75-83: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Use a bounded, cancellation-detached context in the cleanup examples.

context.Background() loses test context values and leaves cleanup API calls unbounded. Show context.WithTimeout(context.WithoutCancel(ctx), ...) instead.

Based on learnings, deferred test cleanup should detach cancellation from the existing context and apply an explicit timeout.

Also applies to: 172-181

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/node/node_mcp_helpers.go` around lines 75 - 83, Update the
deferred CleanupCustomMCP examples around CreateCustomMCPForNode to pass a
context derived from the existing ctx using context.WithoutCancel and an
explicit timeout, rather than context.Background(). Apply the same bounded,
cancellation-detached context pattern to the additional cleanup example
referenced by the comment, preserving deferred cleanup behavior.

Source: Learnings

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/node/additional_storage_e2e.go`:
- Around line 52-60: Deferred cleanup uses contexts that may be canceled or lack
a deadline. In test/extended/node/additional_storage_e2e.go:52-60, create a
bounded context with context.WithTimeout(context.WithoutCancel(ctx), ...) and
use it for cleanup; at 265, avoid passing the potentially canceled spec context
directly. Update the creation example in
test/extended/node/node_mcp_helpers.go:75-83 and cleanup example at 172-181 to
demonstrate the same cancellation-detached, timeout-bounded cleanup pattern.
- Around line 158-197: The test currently validates only configuration and
direct filesystem access, not CRI-O consumption of the layer and artifact
stores. Extend the Phase 4 and Phase 5 flow around the existing storage.conf and
artifact checks to run runtime workloads that resolve content through each
configured store, and assert successful consumption; otherwise narrow the test
name and final assertions to configuration validation rather than functional
store verification.

In `@test/extended/node/node_mcp_helpers.go`:
- Around line 138-151: Make custom MCP setup transactional in
test/extended/node/node_mcp_helpers.go:138-151 by checking rollback errors and
deleting the created MCP plus removing the node label when creation or
WaitForMCP readiness fails; preserve and return the original setup error while
reporting cleanup failures appropriately. In
test/extended/node/node_utils.go:1091-1092, clean up any non-nil configuration
returned by the setup helper before asserting the error.

In `@test/extended/node/node_utils.go`:
- Around line 1122-1127: Return non-NotFound Kubernetes API errors immediately
instead of converting them into polling timeouts: update the pod lookup callback
in test/extended/node/node_utils.go:1122-1127, preserve node lookup errors in
test/extended/node/node_mcp_helpers.go:201-209, and return non-NotFound pod
lookup errors in test/extended/node/additional_storage_e2e.go:248-250. At
test/extended/node/additional_storage_e2e.go:274-285, handle and propagate the
Events List error rather than discarding it; retain successful NotFound polling
behavior.
- Around line 981-985: Update createTestPod in test/extended/node/node_utils.go
to accept and apply an ImagePullPolicy parameter. In
test/extended/node/additional_storage_e2e.go, pass Never for the
prepopulated-image pod at lines 208-215 and Always for the fallback pod at lines
260-268, ensuring each scenario validates its intended image source.

---

Nitpick comments:
In `@test/extended/node/node_mcp_helpers.go`:
- Around line 75-83: Update the deferred CleanupCustomMCP examples around
CreateCustomMCPForNode to pass a context derived from the existing ctx using
context.WithoutCancel and an explicit timeout, rather than context.Background().
Apply the same bounded, cancellation-detached context pattern to the additional
cleanup example referenced by the comment, preserving deferred cleanup behavior.

In `@test/extended/node/node_utils.go`:
- Around line 1174-1265: Remove the duplicated readiness logic from WaitForMCP
and reuse the existing waitForMCP implementation near the earlier MCP wait code
by exporting or wrapping it. Preserve the current options, timeout, draining
behavior, degraded-pool handling, and readiness semantics through that single
implementation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 1dbe9f9f-8f29-447c-8085-45191c385c58

📥 Commits

Reviewing files that changed from the base of the PR and between 478b8a2 and 000f3ae.

📒 Files selected for processing (3)
  • test/extended/node/additional_storage_e2e.go
  • test/extended/node/node_mcp_helpers.go
  • test/extended/node/node_utils.go

Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/node_mcp_helpers.go
Comment thread test/extended/node/node_utils.go Outdated
Comment thread test/extended/node/node_utils.go
Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go
Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go Outdated
@BhargaviGudi
BhargaviGudi force-pushed the additionalImageStore branch 2 times, most recently from 98abffa to a5cb0a3 Compare July 17, 2026 10:45
Comment thread test/extended/node/additional_storage_e2e.go Outdated
@BhargaviGudi
BhargaviGudi force-pushed the additionalImageStore branch 2 times, most recently from b8a6f32 to cfcd6f9 Compare July 17, 2026 11:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/node/additional_storage_e2e.go`:
- Around line 86-89: Update the podman pull execution in the additional
image-store setup to use a context-aware helper with a bounded timeout, rather
than the unbounded ExecOnNodeWithChroot call. Ensure registry or network
failures cancel the command and propagate the timeout error through the existing
assertion.
- Around line 82-95: Update the additional image-store setup for testPod1 to use
a uniquely generated local image tag, and pre-populate that tag only in the
additional store. Ensure the tag is not available in CRI-O’s primary storage
before exercising PullNever, while preserving the existing registry-backed image
reference for the fallback phase.
- Around line 51-59: Update the cleanup registration in the test setup so the
ContainerRuntimeConfig deletion and cleanupSingleNodeMCP rollback execute before
the backing-directory cleanup. Because DeferCleanup runs in LIFO order, register
the directory cleanup first or combine both operations in a single cleanup while
preserving the existing error handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: dab5fff7-2044-43dc-8578-d8923b7111cb

📥 Commits

Reviewing files that changed from the base of the PR and between b8a6f32 and cfcd6f9.

📒 Files selected for processing (4)
  • test/extended/node/README.md
  • test/extended/node/additional_storage_e2e.go
  • test/extended/node/node_mcp_helpers.go
  • test/extended/node/node_utils.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • test/extended/node/README.md
  • test/extended/node/node_mcp_helpers.go
  • test/extended/node/node_utils.go

Comment thread test/extended/node/additional_storage_e2e.go Outdated
Comment thread test/extended/node/additional_storage_e2e.go
Comment thread test/extended/node/additional_storage_e2e.go
@BhargaviGudi
BhargaviGudi force-pushed the additionalImageStore branch from cfcd6f9 to b96eb5d Compare July 17, 2026 12:43
@BhargaviGudi
BhargaviGudi force-pushed the additionalImageStore branch from b96eb5d to df3a6e4 Compare July 17, 2026 12:52
@BhargaviGudi

Copy link
Copy Markdown
Contributor Author

/test all

@BhargaviGudi

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@BhargaviGudi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4608ada0-81df-11f1-93fb-d45cb0af2638-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants