Skip to content

OCPNODE-4494: e2e test case for RHCOS upgrade from 9 → 10#31393

Open
asahay19 wants to merge 1 commit into
openshift:mainfrom
asahay19:rhcos-upgrade
Open

OCPNODE-4494: e2e test case for RHCOS upgrade from 9 → 10#31393
asahay19 wants to merge 1 commit into
openshift:mainfrom
asahay19:rhcos-upgrade

Conversation

@asahay19

@asahay19 asahay19 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the crun happy-path e2e test for RHCOS 9 → 10 osImageStream upgrades, complementing the runc guard test merged in #31266 (UC6A).
On OCP 5.0–5.2 clusters with dual OS streams and crun as the default runtime, moving a pool from rhel-9 to rhel-10 should succeed without triggering the runc-on-RHCOS-10 guard. This test validates that recommended upgrade path.

What the test does

  1. Creates an isolated MachineConfigPool pinned to rhel-9 (no runc ContainerRuntimeConfig)
  2. Labels one pure worker into the pool and waits for rollout on RHCOS 9 with crun
  3. Patches the pool to rhel-10
  4. Asserts:
    • MCP reaches ready without RenderDegraded / Degraded
    • Node reboots to RHCOS 10 and stays Ready
    • Runtime remains crun (no runc CRI-O drop-in)
    • co/machine-config is not Upgradeable=False (reason DegradedPool)

Changes

  • test/extended/node/runc_upgrade_cases.go — new It + helpers; shared AfterEach parameterized by pool name

  • test/extended/node/runc_upgrade_cases.md — documents both UC6A and UC5/UC14

  • Epic: OCPNODE-4013

Locally executed on Openshift cluster 5.0, it got executed successfully:

./openshift-tests run-test \
  "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun"
  
    ------------------------------

  Ran 1 of 1 Specs in 506.229 seconds
  SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[
  {
    "name": "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun",
    "lifecycle": "informing",
    "duration": 506253,
    "startTime": "2026-07-15 12:13:34.449119 UTC",
    "endTime": "2026-07-15 12:22:00.702706 UTC",
    "result": "passed",

Summary by CodeRabbit

  • Tests

    • Expanded end-to-end OS image upgrade coverage to verify RHCOS 9-to-10 upgrades when the default container runtime is crun.
    • Added a new crun “happy path” upgrade scenario and strengthened assertions around rollout health, node reboot behavior, and runtime preservation.
    • Improved and refactored runtime verification and cleanup/rollback handling for both runc guard and crun scenarios.
  • Documentation

    • Updated RHCOS 9-to-10 upgrade test case descriptions, including UC6A formatting and added UC5/UC14 details with run commands and timing/expected-outcome guidance.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 16, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 16, 2026

Copy link
Copy Markdown

@asahay19: This pull request references OCPNODE-4494 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Adds the crun happy-path e2e test for RHCOS 9 → 10 osImageStream upgrades, complementing the runc guard test merged in #31266 (UC6A).
On OCP 5.0–5.2 clusters with dual OS streams and crun as the default runtime, moving a pool from rhel-9 to rhel-10 should succeed without triggering the runc-on-RHCOS-10 guard. This test validates that recommended upgrade path.

What the test does

  1. Creates an isolated MachineConfigPool pinned to rhel-9 (no runc ContainerRuntimeConfig)
  2. Labels one pure worker into the pool and waits for rollout on RHCOS 9 with crun
  3. Patches the pool to rhel-10
  4. Asserts:
  • MCP reaches ready without RenderDegraded / Degraded
  • Node reboots to RHCOS 10 and stays Ready
  • Runtime remains crun (no runc CRI-O drop-in)
  • co/machine-config is not Upgradeable=False (reason DegradedPool)

Changes

  • test/extended/node/runc_upgrade_cases.go — new It + helpers; shared AfterEach parameterized by pool name

  • test/extended/node/runc_upgrade_cases.md — documents both UC6A and UC5/UC14

  • Epic: OCPNODE-4013

Locally executed on Openshift cluster 5.0, it got executed successfully:

./openshift-tests run-test \
 "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun"
 
   ------------------------------

 Ran 1 of 1 Specs in 506.229 seconds
 SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[
 {
   "name": "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun",
   "lifecycle": "informing",
   "duration": 506253,
   "startTime": "2026-07-15 12:13:34.449119 UTC",
   "endTime": "2026-07-15 12:22:00.702706 UTC",
   "result": "passed",

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: asahay19
Once this PR has been reviewed and has the lgtm label, please assign rphillips for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
openshift-ci Bot requested review from bitoku and sairameshv July 16, 2026 05:29
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 78482217-91d1-4ddc-bb29-b0118accdba3

📥 Commits

Reviewing files that changed from the base of the PR and between dfe3764 and c6aab91.

📒 Files selected for processing (2)
  • test/extended/node/runc_upgrade_cases.go
  • test/extended/node/runc_upgrade_cases.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/extended/node/runc_upgrade_cases.md
  • test/extended/node/runc_upgrade_cases.go

Walkthrough

The upgrade test suite now covers both runc guard behavior and the crun happy path for RHCOS 9-to-10 upgrades. Cleanup is parameterized by the active test pool, and documentation describes both scenarios and their health assertions.

Changes

Runtime upgrade test coverage

Layer / File(s) Summary
Runtime scenarios and pool cleanup
test/extended/node/runc_upgrade_cases.go
Adds the crun upgrade pool and validates its RHCOS 9-to-10 path, while parameterizing cleanup for the active pool and optional CRC removal.
Runtime and readiness helpers
test/extended/node/runc_upgrade_cases.go
Separates polling helpers for runc and crun runtime checks from immediate node readiness assertions, and adds MCP and upgradeability assertions.
Upgrade case documentation
test/extended/node/runc_upgrade_cases.md
Documents the runc guard and crun happy-path cases, commands, condition timing, and separate runtime estimates.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant UpgradeTest
  participant MachineConfigPool
  participant WorkerNode
  participant ClusterOperator
  UpgradeTest->>MachineConfigPool: Create crun upgrade pool
  UpgradeTest->>WorkerNode: Label worker and verify RHCOS 9 with crun
  UpgradeTest->>MachineConfigPool: Set osImageStream to rhel-10
  MachineConfigPool->>WorkerNode: Roll out RHCOS 10 configuration
  UpgradeTest->>MachineConfigPool: Assert no RenderDegraded
  UpgradeTest->>ClusterOperator: Check DegradedPool does not block upgradeability
Loading

Suggested reviewers: sairameshv


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The patch adds several framework.Logf messages that print nodeName, which can expose internal hostnames in test logs. Redact or hash node identifiers in log messages, or remove nodeName from logs unless strictly necessary for debugging.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: an end-to-end test for RHCOS 9→10 upgrade behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The Ginkgo suite and It() names are fixed strings; no dynamic IDs, node names, dates, UUIDs, or generated suffixes appear in the titles.
Test Structure And Quality ✅ Passed Both It blocks target one scenario, cluster resources are cleaned via AfterEach, all cluster waits have explicit timeouts, and assertions are mostly context-rich.
Microshift Test Compatibility ✅ Passed The new crun test is inside a suite that skips MicroShift in BeforeEach, so its OpenShift-only APIs/features are not exercised there.
Single Node Openshift (Sno) Test Compatibility ✅ Passed Shared BeforeEach skips configv1.SingleReplicaTopologyMode before both new It() cases, so the added tests are not expected to run on SNO.
Topology-Aware Scheduling Compatibility ✅ Passed Only e2e test/docs changed; no operator/manifests/controllers or topology-sensitive scheduling logic was introduced.
Ote Binary Stdout Contract ✅ Passed The PR adds only test-case code and cleanup callbacks; no process-level main/init/suite code or stdout prints/log redirection were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed New Ginkgo tests only use cluster APIs and node exec; I found no hardcoded IPv4, IPv4-only parsing, or public internet/external-host dependencies.
No-Weak-Crypto ✅ Passed The PR only adds/renames MCO e2e test helpers and docs; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons appear in the diff.
Container-Privileges ✅ Passed The PR only changes a Go e2e test and markdown; no manifest or securityContext changes introduce privileged, host* namespaces, SYS_ADMIN, or allowPrivilegeEscalation.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 16, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@asahay19: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn 6b176d2 link true /test e2e-gcp-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@bitoku bitoku left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lgtm,
One comment about the style.
There's some functions like assert*, wait*, expect*, verify*.
I don't mind either, but want them to be consistent or follow a rule (for example, expect calls Expect in the function, something like that).

Comment on lines +62 to +64
nodeName = ""
testPoolName = ""
cleanupCRC = false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't think they are needed. They should be initialized when it's defined.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped redundant BeforeEach resets for nodeName / testPoolName / cleanupCRC in the latest commit


g.By("Upgrading RHCOS version to RHCOS 10 via osImageStream")
o.Expect(setPoolOSImageStream(ctx, mcClient, crunRHCOS10UpgradePool, streamRHEL10)).To(o.Succeed())
o.Expect(waitForMCP(ctx, mcClient, crunRHCOS10UpgradePool, 45*time.Minute)).To(o.Succeed(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think 45 mins is too long. It's supposed to be finished much earlier.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced post-rhel-10 MCP and node waits from 45m to 30m

"pool should roll out to rhel-10 without runc guard RenderDegraded")

g.By("Verifying node rolled out to RHCOS 10 with crun and pool stayed healthy")
o.Expect(waitForNodeRHELMajorVersion(ctx, oc, nodeName, "10", 45*time.Minute)).To(o.Succeed(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think 45 mins is too long. It's supposed to be finished much earlier.

if upgradeable != nil &&
upgradeable.Status == configv1.ConditionFalse &&
upgradeable.Reason == degradedPoolUpgradeableReason {
return fmt.Errorf("ClusterOperator %s Upgradeable=False (reason %s) after crun rhel-10 upgrade: %q",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too specific error message, this may be confusing in the future when someone uses this function.

Suggested change
return fmt.Errorf("ClusterOperator %s Upgradeable=False (reason %s) after crun rhel-10 upgrade: %q",
return fmt.Errorf("ClusterOperator %s Upgradeable=False (reason %s): %q",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generalized assertMachineConfigNotBlockedByDegradedPool error message in the latest commit

@bitoku

bitoku commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2 periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@bitoku: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/07d107d0-8112-11f1-980f-e0e7979d9599-0

@asahay19

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@asahay19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/bc3277f0-81d4-11f1-91a5-e78e2058153d-0

@asahay19

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@asahay19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/cab5adb0-81d4-11f1-9135-6cfa88222756-0

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@asahay19

Copy link
Copy Markdown
Contributor Author

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants