Repository navigation
feat: operator upgrades - #368
Conversation
Reviewer's GuideImplements preliminary operator upgrade support by recording the reconciled operator version, detecting drift in the TEC reconcile loop, converging Trustee and related images through staged readiness-checked rollouts, invalidating approved-image PCR state for recomputation, and validating both successful and failed real-version upgrades with integration tests and updated documentation. Sequence diagram for the operator upgrade reconciliationsequenceDiagram
participant TEC as TrustedExecutionCluster
participant Operator
participant Trustee as TrusteeDeployment
participant Related as RelatedImageDeployments
participant Images as ApprovedImages
participant Status as TECStatus
Operator->>TEC: reconcile()
Operator->>Status: Read observedOperatorVersion
alt version drift detected
Operator->>Status: Set Upgrade=InProgress
Operator->>Trustee: converge_trustee()
Trustee->>Trustee: apply_kbs_deployment()
Operator->>Trustee: wait_for_deployment_available()
Operator->>Images: invalidate_all_approved_images()
Images->>Images: delete_compute_pcrs_job()
Images->>Images: Recompute PCR state
Operator->>Trustee: trustee_deployment_reconcile()
Operator->>Related: converge_related_images()
Related->>Related: Patch deployment images
Operator->>Related: wait_for_deployment_available()
Operator->>Status: Set stage conditions Complete
Operator->>Status: Set Upgrade=Complete and observedOperatorVersion
else previous upgrade failed
Operator->>Status: Keep Upgrade=Failed
end
State diagram for operator upgrade statusstateDiagram-v2
[*] --> Installed
Installed --> UpgradeInProgress: observedOperatorVersion != COMPONENT_VERSION
UpgradeInProgress --> TrusteeUpgrade: run_upgrade()
TrusteeUpgrade --> RelatedImagesUpgrade: converge_trustee() succeeds
RelatedImagesUpgrade --> UpgradeComplete: converge_related_images() succeeds
TrusteeUpgrade --> UpgradeFailed: stage fails
RelatedImagesUpgrade --> UpgradeFailed: stage fails
UpgradeComplete --> Installed: observedOperatorVersion updated
UpgradeFailed --> UpgradeFailed: manual intervention required
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've reviewed your changes and they look great!
Sourcery assessment
Needs a human reviewer. This changes live Kubernetes Deployments and Trustee configuration, invalidates approved-image PCR state, and can leave an upgrade partially applied if rollout or synchronization fails. Reverting the operator does not automatically restore changed images, status, deleted PCR Jobs, or any already-disrupted attestation service, although the affected state is generally bounded and repairable.
3c593b7 to
e0ab491
Compare
Jakob-Naucke
left a comment
There was a problem hiding this comment.
random first comments:
- can the test go in a separate commit?
- please format your commit messages according to CONTRIBUTING.md (50 lines subject, 72 body)
Yes, I will do both of these, but I'm not sure of how to split any further without either git bisects failing or cargo checks failing on each commit. |
e0ab491 to
324c972
Compare
|
/test azure-integration-tests |
|
/test azure-integration-test |
Jakob-Naucke
left a comment
There was a problem hiding this comment.
some comments on the first commit
324c972 to
34560c4
Compare
Jakob-Naucke
left a comment
There was a problem hiding this comment.
some first random comments on the tests. sorry that's all for today, the commit is huge and hard to penetrate
Jakob-Naucke
left a comment
There was a problem hiding this comment.
sorry, still couldn't cover everything I wanted but sending this for now
354090e to
683851c
Compare
c672109 to
345a87f
Compare
|
I couldn't get a clean split at the files in which we modify existing unit tests, hence the changes to existing unit tests are in the dev commit itself, and new unit tests are in the unit tests commit. This ensures bisects work, and |
|
There is some issue with the rebase, changes from previous commits part of main have leaked here. I am working on fixing it. edit: fixed. |
|
/test azure-integration-test |
|
@SpaceFace02: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
345a87f to
ca79acb
Compare
Jakob-Naucke
left a comment
There was a problem hiding this comment.
Sorry, still could not get to the integration tests commits today but I should be able tomorrow and maybe I don't have too many comments on them
67af693 to
55c2433
Compare
There was a problem hiding this comment.
Commit: "feat: Add operator upgrade reconciliation"
The Trustee Deployment gets a RollingUpdate strategy ...
When the new image never becomes ready and the old pod keeps serving while the ConfigMap/Deployment are now patched - there is a version-mismatch between the (new) ConfigMap and the (old) Operator.
Yes, In that case, the |
f6de31d to
23d79b9
Compare
|
/azure-integration-test |
|
/test azure-integration-test |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Jakob-Naucke, SpaceFace02 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
@uril Do let me know if I answered your question, or if any change is required |
Add a generic has_condition<S: Conditions> to trusted-cluster-operator-lib that checks a status for a condition type/reason, usable for any status type implementing Conditions. Also grouped various upgrade conditions with TEC. Signed-off-by: Chirag Rao <crao@redhat.com>
Reconcile now converges the Trustee and related-image Deployments to the versions pinned by RELATED_IMAGE_* env vars whenever the observed operator version drifts from the running binary, tracked via a new observedOperatorVersion status field. The Trustee Deployment gets a RollingUpdate strategy (maxUnavailable 0) so an old pod keeps serving if the new image never becomes ready, facilitating zero downtime, and its ConfigMap/Deployment are now patched in place during upgrade (converge_trustee reuses install_trustee_configuration instead of duplicating it). ApprovedImages are invalidated to force PCR recomputation whenever upgrade runs. Reconcile-phase handlers take a lightweight ReconcileState (name, generation, existing_status) plus the shared OperatorContext and building ApiContext wherever required. Assisted-by: AI Signed-off-by: Chirag Rao <crao@redhat.com>
Adds unit coverage for the new upgrade conditions and reconcile paths for operator upgrades. Assisted-by: AI Signed-off-by: Chirag Rao <crao@redhat.com>
Extends the trusted_execution_cluster integration suite with coverage for the upgrade flow: a forced related-image upgrade, RelatedImagesUpgrade condition transitions, and a failed-upgrade scenario that leaves Upgrade=Failed without retrying. Assisted-by: AI Signed-off-by: Chirag Rao <crao@redhat.com>
Adds an integration suite (tests/upgrade.rs, `make upgrade-tests`) for operator upgrades/ When the new manifests are applied, this test file ensures upgrades happen smoothly from one operator source version to another. Assisted-by: AI Signed-off-by: Chirag Rao <crao@redhat.com>
Updates the documentation to highlight that pcr values are now stored as part of approvedImage status field itself, instead of image-pcrs configmap. The image-pcrs configmap is removed entirely from the operator's architecture. Also updates the documentation to highlight operator changes as per trustee version 0.20.0, where reference value configmap was removed in favour of ephemeral reference values stored in trustee pod. Also modifies rv-flow.png and image-flow.png to reflect this new change. Signed-off-by: Chirag Rao <crao@redhat.com>
Adds a user guide and design doc and architecture diagrams for the operator upgrade flow. Signed-off-by: Chirag Rao <crao@redhat.com>
23d79b9 to
30ee948
Compare
|
New changes are detected. LGTM label has been removed. |
9af0f88
into
trusted-execution-clusters:main
This PR adds a preliminary operator upgrades support. The operator's reconcile loop will attempt to converge all dependant images on an upgrade, either manually applied or via OLM. It also includes integration tests for the same, testing a real version upgrade, from a old tag on quay, to your locally built artifacts and binaries.
Please refer the docs updated for more details.
Summary by Sourcery
Implement operator upgrades that detect version drift, converge managed components, recompute image reference values, and report recoverable upgrade failures.
New Features:
Bug Fixes:
Enhancements:
Build:
Documentation:
Tests:
Chores: