Skip to content

feat: operator upgrades - #368

Merged
SpaceFace02 merged 7 commits into
trusted-execution-clusters:mainfrom
SpaceFace02:operator-upgrades
Oct 27, 2026
Merged

SpaceFace02 merged 7 commits into
trusted-execution-clusters:mainfrom
SpaceFace02:operator-upgrades

Conversation

@SpaceFace02

@SpaceFace02 SpaceFace02 commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

This PR adds a preliminary operator upgrades support. The operator's reconcile loop will attempt to converge all dependant images on an upgrade, either manually applied or via OLM. It also includes integration tests for the same, testing a real version upgrade, from a old tag on quay, to your locally built artifacts and binaries.

Please refer the docs updated for more details.

Summary by Sourcery

Implement operator upgrades that detect version drift, converge managed components, recompute image reference values, and report recoverable upgrade failures.

New Features:

  • Add automatic operator upgrade handling that detects version drift and converges Trustee and related component images through reconciliation.
  • Store operator version and upgrade-stage status in TrustedExecutionCluster status, including progress and failure reporting.
  • Recompute ApprovedImage PCR data during upgrades and synchronize the resulting reference values with Trustee.
  • Add end-to-end upgrade coverage for successful real-version upgrades and failed upgrades that preserve running components and attestation.

Bug Fixes:

  • Preserve service availability during Trustee upgrades by waiting for rollouts and retaining the existing deployment during failed updates.
  • Move PCR and reference-value state from the shared ConfigMap into ApprovedImage status and update Trustee synchronization accordingly.

Enhancements:

  • Add shared rollout, deletion, condition, and image-management helpers for reconciliation and integration tests.
  • Support configurable deployment readiness timeouts and component image overrides for upgrade testing.

Build:

  • Add Makefile configuration and targets for upgrade integration tests.

Documentation:

  • Document the operator upgrade design, user upgrade procedure, limitations, monitoring, and failure recovery.
  • Update reference-value and operator architecture documentation to describe ApprovedImage status and Trustee API synchronization.

Tests:

  • Add unit tests for upgrade conditions, version drift, failure handling, image invalidation, rollout behavior, and TPM event coverage.
  • Add integration tests covering successful upgrades, PCR recomputation, component convergence, failed upgrades, and post-upgrade attestation.

Chores:

  • Improve integration-test cleanup and finalizer verification for attestation-key Secrets.

@sourcery-ai

sourcery-ai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implements preliminary operator upgrade support by recording the reconciled operator version, detecting drift in the TEC reconcile loop, converging Trustee and related images through staged readiness-checked rollouts, invalidating approved-image PCR state for recomputation, and validating both successful and failed real-version upgrades with integration tests and updated documentation.

Sequence diagram for the operator upgrade reconciliation

sequenceDiagram
    participant TEC as TrustedExecutionCluster
    participant Operator
    participant Trustee as TrusteeDeployment
    participant Related as RelatedImageDeployments
    participant Images as ApprovedImages
    participant Status as TECStatus

    Operator->>TEC: reconcile()
    Operator->>Status: Read observedOperatorVersion
    alt version drift detected
        Operator->>Status: Set Upgrade=InProgress
        Operator->>Trustee: converge_trustee()
        Trustee->>Trustee: apply_kbs_deployment()
        Operator->>Trustee: wait_for_deployment_available()
        Operator->>Images: invalidate_all_approved_images()
        Images->>Images: delete_compute_pcrs_job()
        Images->>Images: Recompute PCR state
        Operator->>Trustee: trustee_deployment_reconcile()
        Operator->>Related: converge_related_images()
        Related->>Related: Patch deployment images
        Operator->>Related: wait_for_deployment_available()
        Operator->>Status: Set stage conditions Complete
        Operator->>Status: Set Upgrade=Complete and observedOperatorVersion
    else previous upgrade failed
        Operator->>Status: Keep Upgrade=Failed
    end
Loading

State diagram for operator upgrade status

stateDiagram-v2
    [*] --> Installed
    Installed --> UpgradeInProgress: observedOperatorVersion != COMPONENT_VERSION
    UpgradeInProgress --> TrusteeUpgrade: run_upgrade()
    TrusteeUpgrade --> RelatedImagesUpgrade: converge_trustee() succeeds
    RelatedImagesUpgrade --> UpgradeComplete: converge_related_images() succeeds
    TrusteeUpgrade --> UpgradeFailed: stage fails
    RelatedImagesUpgrade --> UpgradeFailed: stage fails
    UpgradeComplete --> Installed: observedOperatorVersion updated
    UpgradeFailed --> UpgradeFailed: manual intervention required
Loading

File-Level Changes

Change Details Files
Add version-drift detection and a staged upgrade workflow to the main reconciliation loop.
  • Track the last successfully reconciled operator version in status.
  • Detect upgrades separately from fresh installation and persist InProgress, Complete, or Failed conditions.
  • Stop automatic retries after a failed upgrade until manual intervention.
api/v1alpha1/crds.go
api/v1alpha1/conditions.go
lib/src/conditions.rs
operator/src/conditions.rs
operator/src/main.rs
Converge dependent workloads safely during upgrades while preserving service availability.
  • Patch Trustee configuration and Deployment in place, using a zero-unavailable rolling update and readiness timeout.
  • Update related-image Deployments only when image drift is detected, waiting for each rollout.
  • Resync Trustee API state after rollout and repair labels on secrets created by older operator versions.
  • Skip unnecessary Trustee updates during TEC deletion cleanup.
operator/src/main.rs
operator/src/trustee.rs
operator/src/attestation_key_register.rs
Invalidate and recompute approved-image PCR state after Trustee upgrades.
  • Delete existing PCR computation Jobs and reset ApprovedImage commitment state while preserving relevant metadata.
  • Add a TPM event parsing coverage test to detect newly introduced library variants.
operator/src/main.rs
operator/src/reference_values.rs
lib/src/reference_values.rs
Add documentation and configuration for upgrade testing and behavior.
  • Document additive CRD constraints, reconciliation-based upgrades, PCR recomputation, and manual recovery requirements.
  • Add configurable old registry and version tags plus a dedicated upgrade-tests Make target.
docs/design/operator_upgrades.md
Makefile
Add unit, mock-client, and end-to-end integration coverage for upgrade scenarios.
  • Test same-version no-op, successful version drift, failure status persistence, PCR invalidation, rollout readiness, and Trustee ConfigMap update behavior.
  • Add a real old-tag-to-local-artifact upgrade test covering component convergence, PCR/event migration, VM re-attestation, and failed-upgrade availability.
  • Extend test utilities with image overrides, rollout polling, upgrade triggers, status predicates, and cleanup checks.
operator/src/main.rs
operator/src/conditions.rs
operator/src/trustee.rs
test_utils/src/lib.rs
tests/Cargo.toml
tests/trusted_execution_cluster.rs
tests/upgrade.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Needs a human reviewer. This changes live Kubernetes Deployments and Trustee configuration, invalidates approved-image PCR state, and can leave an upgrade partially applied if rollout or synchronization fails. Reverting the operator does not automatically restore changed images, status, deleted PCR Jobs, or any already-disrupted attestation service, although the affected state is generally bounded and repairable.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

@SpaceFace02
SpaceFace02 force-pushed the operator-upgrades branch 3 times, most recently from 3c593b7 to e0ab491 Compare September 15, 2026 07:34

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

random first comments:

  • can the test go in a separate commit?
  • please format your commit messages according to CONTRIBUTING.md (50 lines subject, 72 body)

@SpaceFace02

SpaceFace02 commented Sep 15, 2026 •

Copy link
Copy Markdown
Member Author

can the test go in a separate commit?
please format your commit messages according to CONTRIBUTING.md (50 lines subject, 72 body)

Yes, I will do both of these, but I'm not sure of how to split any further without either git bisects failing or cargo checks failing on each commit.

@SpaceFace02

Copy link
Copy Markdown
Member Author

/test azure-integration-tests

@SpaceFace02

Copy link
Copy Markdown
Member Author

/test azure-integration-test

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some comments on the first commit

Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some first random comments on the tests. sorry that's all for today, the commit is huge and hard to penetrate

Comment thread operator/src/conditions.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, still couldn't cover everything I wanted but sending this for now

Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
@SpaceFace02
SpaceFace02 force-pushed the operator-upgrades branch 2 times, most recently from 354090e to 683851c Compare September 21, 2026 07:43
@SpaceFace02
SpaceFace02 requested a review from uril September 22, 2026 09:14

@Jakob-Naucke Jakob-Naucke left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full (I hope) review of the first commit. Sorry for catching many things only now.

Comment thread api/v1alpha1/conditions.go Resolved
Comment thread lib/src/conditions.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/reference_values.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread tests/trusted_execution_cluster.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
@SpaceFace02
SpaceFace02 force-pushed the operator-upgrades branch 2 times, most recently from c672109 to 345a87f Compare October 9, 2026 10:49
@SpaceFace02

Copy link
Copy Markdown
Member Author

I couldn't get a clean split at the files in which we modify existing unit tests, hence the changes to existing unit tests are in the dev commit itself, and new unit tests are in the unit tests commit. This ensures bisects work, and cargo check passes for each commit.

@SpaceFace02

SpaceFace02 commented Oct 9, 2026 •

Copy link
Copy Markdown
Member Author

There is some issue with the rebase, changes from previous commits part of main have leaked here. I am working on fixing it.

edit: fixed.

@SpaceFace02

Copy link
Copy Markdown
Member Author

/test azure-integration-test

@openshift-ci

openshift-ci Bot commented Oct 9, 2026

Copy link
Copy Markdown

@SpaceFace02: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/azure-integration-test 345a87f link false /test azure-integration-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, still could not get to the integration tests commits today but I should be able tomorrow and maybe I don't have too many comments on them

Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread test_utils/src/lib.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/main.rs Resolved
Comment thread operator/src/reference_values.rs Resolved
Comment thread tests/trusted_execution_cluster.rs Resolved
Comment thread tests/trusted_execution_cluster.rs Resolved
Comment thread tests/trusted_execution_cluster.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread tests/upgrade.rs Resolved
Comment thread docs/design/operator-architecture.md Resolved
Comment thread docs/design/operator_upgrades.md Resolved
Comment thread docs/design/operator_upgrades.md Resolved
Comment thread docs/usage/getting-started-guide.md Resolved
Comment thread docs/usage/getting-started-guide.md Resolved
@SpaceFace02
SpaceFace02 force-pushed the operator-upgrades branch 2 times, most recently from 67af693 to 55c2433 Compare October 15, 2026 08:09

@uril uril left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit: "feat: Add operator upgrade reconciliation"

The Trustee Deployment gets a RollingUpdate strategy ...

When the new image never becomes ready and the old pod keeps serving while the ConfigMap/Deployment are now patched - there is a version-mismatch between the (new) ConfigMap and the (old) Operator.

@SpaceFace02

SpaceFace02 commented Oct 19, 2026 •

Copy link
Copy Markdown
Member Author

When the new image never becomes ready and the old pod keeps serving while the ConfigMap/Deployment are now patched - there is a version-mismatch between the (new) ConfigMap and the (old) Operator.

Yes, In that case, the Upgrade=Failed condition will be set, so admin can take necessary action. This indicates cluster is in a undesired shape and must be rectified. Don't know what would be the cleanest way to handle such a a case in the future

@SpaceFace02
SpaceFace02 force-pushed the operator-upgrades branch 2 times, most recently from f6de31d to 23d79b9 Compare October 19, 2026 10:03
@SpaceFace02

Copy link
Copy Markdown
Member Author

/azure-integration-test

@SpaceFace02

Copy link
Copy Markdown
Member Author

/test azure-integration-test

@openshift-ci

openshift-ci Bot commented Oct 26, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jakob-Naucke, SpaceFace02

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@SpaceFace02

Copy link
Copy Markdown
Member Author

When the new image never becomes ready and the old pod keeps serving while the ConfigMap/Deployment are now patched - there is a version-mismatch between the (new) ConfigMap and the (old) Operator.

@uril Do let me know if I answered your question, or if any change is required

Add a generic has_condition<S: Conditions> to trusted-cluster-operator-lib
that checks a status for a condition type/reason, usable for any status
type implementing Conditions.

Also grouped various upgrade conditions with TEC.

Signed-off-by: Chirag Rao <crao@redhat.com>
Reconcile now converges the Trustee and related-image Deployments
to the versions pinned by RELATED_IMAGE_* env vars whenever the
observed operator version drifts from the running binary, tracked
via a new observedOperatorVersion status field.

The Trustee Deployment gets a RollingUpdate strategy (maxUnavailable
0) so an old pod keeps serving if the new image never becomes ready,
facilitating zero downtime, and its ConfigMap/Deployment are now
patched in place during upgrade (converge_trustee reuses
install_trustee_configuration instead of duplicating it).
ApprovedImages are invalidated to force PCR recomputation whenever
upgrade runs.

Reconcile-phase handlers take a lightweight ReconcileState (name,
generation, existing_status) plus the shared OperatorContext and building ApiContext wherever required.

Assisted-by: AI
Signed-off-by: Chirag Rao <crao@redhat.com>
Adds unit coverage for the new upgrade conditions and reconcile
paths for operator upgrades.

Assisted-by: AI
Signed-off-by: Chirag Rao <crao@redhat.com>
Extends the trusted_execution_cluster integration suite with
coverage for the upgrade flow: a forced related-image upgrade,
RelatedImagesUpgrade condition transitions, and a failed-upgrade
scenario that leaves Upgrade=Failed without retrying.

Assisted-by: AI
Signed-off-by: Chirag Rao <crao@redhat.com>
Adds an integration suite (tests/upgrade.rs, `make upgrade-tests`) for operator upgrades/
When the new manifests are applied, this test file ensures upgrades happen smoothly from one operator source version to another.

Assisted-by: AI
Signed-off-by: Chirag Rao <crao@redhat.com>
Updates the documentation to highlight that pcr values are now stored as part of approvedImage status field itself, instead of image-pcrs configmap.
The image-pcrs configmap is removed entirely from the operator's architecture.
Also updates the documentation to highlight operator changes as per trustee version 0.20.0, where reference value configmap was removed in favour of ephemeral reference values stored in trustee pod.

Also modifies rv-flow.png and image-flow.png to reflect this new change.

Signed-off-by: Chirag Rao <crao@redhat.com>
Adds a user guide and design doc and architecture diagrams for the operator upgrade flow.

Signed-off-by: Chirag Rao <crao@redhat.com>
@openshift-ci

openshift-ci Bot commented Oct 27, 2026

Copy link
Copy Markdown

New changes are detected. LGTM label has been removed.

@SpaceFace02
SpaceFace02 merged commit 9af0f88 into trusted-execution-clusters:main Oct 27, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants