Repository navigation
Create ApprovedImages based on MachineConfigs - #376
Conversation
Reviewer's GuideIntroduces optional OpenShift integration that watches labeled MachineConfigPools and their selected MachineConfigs, creates deterministic ApprovedImages for eligible osImageURLs, and wires the feature through generated APIs, reconciliation, RBAC, CI, tests, packaging, and documentation while centralizing resource naming and creation utilities. Sequence diagram for OpenShift MachineConfig ApprovedImage creationsequenceDiagram
participant MCP as MachineConfigPool
participant MC as MachineConfig
participant Controller as OpenShiftImageController
participant Store as ApprovedImageStore
participant API as KubernetesAPI
MCP->>Controller: watch labeled MachineConfigPool
Controller->>MC: select matching MachineConfigs
MC-->>Controller: osImageURL
Controller->>Store: check matching ApprovedImage
alt eligible image is not stored
Controller->>API: create ApprovedImage
API-->>Controller: ApprovedImage created
else image already exists
Controller-->>Controller: requeue reconciliation
end
Flow diagram for OpenShift MachineConfig eligibilityflowchart TD
A[MachineConfigPool] --> B{Has autocreate label?}
B -- No --> X[Ignore MachineConfigPool]
B -- Yes --> C[Select MachineConfigs by labels]
C --> D{MachineConfig has osImageURL?}
D -- No --> Y[Skip image creation]
D -- Yes --> E[Generate deterministic RFC1035 name]
E --> F[Create ApprovedImage]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 3 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="lib/src/reference_values.rs" line_range="23" />
<code_context>
+ let hash = hash(MessageDigest::sha1(), name.as_bytes())?;
+ let hashed = hex::encode(hash)[..10].to_string();
+ let formatted = format!("{prefix}-{hashed}-{replaced}");
+ Ok(formatted[..63].trim_end_matches('-').to_string())
+}
</code_context>
<issue_to_address>
**issue (bug_risk):** `formatted[..63]` panics when the generated name is shorter than 63 bytes, so `rfc1035("x", "prefix")` crashes instead of returning a valid truncated name.
**Triggers:** When a caller passes a short image reference or resource name.
**Suggested fix:** Use a bounded truncation operation such as `formatted.chars().take(63)` or slice only up to `formatted.len()`.
```suggestion
Ok(formatted.chars().take(63).collect::<String>().trim_end_matches('-').to_string())
```
</issue_to_address>
### Comment 2
<location path="operator/src/mc_images.rs" line_range="37-48" />
<code_context>
+ return Ok(LONG_REQUEUE);
+ };
+
+ let image_name = rfc1035(&format!("{mcp_name}-{url}"), OSIMAGE_RESOURCE_PREFIX)?;
+ let labels = BTreeMap::from([(MCP_LABEL.to_string(), mcp_name.to_string())]);
+ let image = ApprovedImage {
+ metadata: ObjectMeta {
+ name: Some(image_name),
+ labels: Some(labels),
+ ..Default::default()
+ },
+ spec: ApprovedImageSpec { image: url },
+ status: None,
+ };
+ create_or_info_if_exists(ctx.client.clone(), &image).await?;
+ Ok(LONG_REQUEUE)
+}
</code_context>
<issue_to_address>
**issue (bug_risk):** When an MCP changes to reference a different MachineConfig or OS image, `add_approved_image` creates a new ApprovedImage but never deletes the previous one. On MCP deletion, `delete_approved_image` also deletes only the first matching image returned by the store, leaving any older images with the same MCP label behind.
**Triggers:** When an MCP configuration or its OS image changes, or when an MCP has accumulated more than one generated ApprovedImage.
**Suggested fix:** Track the current generated image identity and delete superseded images during apply; during cleanup, list and delete every ApprovedImage carrying the MCP label.
</issue_to_address>
### Comment 3
<location path="operator/Cargo.toml" line_range="39" />
<code_context>
jsonwebtoken-openssl = "2.0.0"
+[features]
+openshift = ["trusted-cluster-operator-lib/openshift"]
+
[dev-dependencies]
</code_context>
<issue_to_address>
**issue (testing):** Enabling the operator's `openshift` feature does not enable the separate `trusted-cluster-operator-test-utils/openshift` feature, so `#[cfg(not(feature = "openshift"))]` remains active in test_utils and tests still apply ApprovedImage manifests.
**Triggers:** When tests are run with the operator's `openshift` feature enabled.
**Suggested fix:** Forward the feature to test_utils in the relevant dependency feature definition, and define/propagate the feature for the integration-tests crate as well.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 3 findings to address first, and the operator now creates and deletes persisted ApprovedImage resources from cluster-wide MachineConfigPool state; a wrong association or image URL could cause PCR computation and attestation decisions to use the wrong image, while deletion can remove existing approval records. Reverting stops the new controller but does not automatically restore resources it created or deleted, so recovery requires reconciliation or manual repair.
Blocking findings: lib/src/reference_values.rs:23, operator/src/mc_images.rs:48, operator/Cargo.toml:39
| let image_name = rfc1035(&format!("{mcp_name}-{url}"), OSIMAGE_RESOURCE_PREFIX)?; | ||
| let labels = BTreeMap::from([(MCP_LABEL.to_string(), mcp_name.to_string())]); | ||
| let image = ApprovedImage { | ||
| metadata: ObjectMeta { | ||
| name: Some(image_name), | ||
| labels: Some(labels), | ||
| ..Default::default() | ||
| }, | ||
| spec: ApprovedImageSpec { image: url }, | ||
| status: None, | ||
| }; | ||
| create_or_info_if_exists(ctx.client.clone(), &image).await?; |
There was a problem hiding this comment.
issue (bug_risk): When an MCP changes to reference a different MachineConfig or OS image, add_approved_image creates a new ApprovedImage but never deletes the previous one. On MCP deletion, delete_approved_image also deletes only the first matching image returned by the store, leaving any older images with the same MCP label behind.
Triggers: When an MCP configuration or its OS image changes, or when an MCP has accumulated more than one generated ApprovedImage.
Suggested fix: Track the current generated image identity and delete superseded images during apply; during cleanup, list and delete every ApprovedImage carrying the MCP label.
59caf03 to
f7dbb9a
Compare
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="operator/src/mc_images.rs" line_range="154-157" />
<code_context>
+ let mcp_ctx = ctx.clone();
+ tokio::spawn(
+ Controller::new(mcs, Default::default())
+ .watches(mcps, Default::default(), move |mcp| {
+ let mcs = mcp_ctx.mc_store.state_filter(|mc| is_live(&mcp, mc));
+ mcs.into_iter()
+ .filter_map(|mc| mc.metadata.name.as_deref().map(ObjectRef::new))
+ .collect::<Vec<_>>()
+ })
</code_context>
<issue_to_address>
**issue (broader_impact):** When a MachineConfigPool selector is narrowed or otherwise changed, the watch mapper enqueues only MachineConfigs matching the new selector. MachineConfigs that matched the old selector but no longer match are never reconciled, so their ApprovedImages remain after the MachineConfigPool stops selecting them.
**Triggers:** When an existing MachineConfigPool changes its selector to exclude previously selected MachineConfigs.
**Suggested fix:** Enqueue the union of MachineConfigs matching the old and new pool selectors, or reconcile all MachineConfigs affected by the pool event before applying the new scope.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 1 finding to address first, and the new controller automatically creates and deletes ApprovedImages from OpenShift MachineConfigPool selectors, changing which OS images are trusted for attestation and potentially revoking images through the downstream Trustee flow. Reverting the operator does not restore ApprovedImages or trust data that were already created or deleted, so a mistaken selector or cleanup decision requires separate repair.
Blocking findings: operator/src/mc_images.rs:157
| .watches(mcps, Default::default(), move |mcp| { | ||
| let mcs = mcp_ctx.mc_store.state_filter(|mc| is_live(&mcp, mc)); | ||
| mcs.into_iter() | ||
| .filter_map(|mc| mc.metadata.name.as_deref().map(ObjectRef::new)) |
There was a problem hiding this comment.
issue (broader_impact): When a MachineConfigPool selector is narrowed or otherwise changed, the watch mapper enqueues only MachineConfigs matching the new selector. MachineConfigs that matched the old selector but no longer match are never reconciled, so their ApprovedImages remain after the MachineConfigPool stops selecting them.
Triggers: When an existing MachineConfigPool changes its selector to exclude previously selected MachineConfigs.
Suggested fix: Enqueue the union of MachineConfigs matching the old and new pool selectors, or reconcile all MachineConfigs affected by the pool event before applying the new scope.
There was a problem hiding this comment.
I'm not sure the previous state of the MCP is really available here. I'm tempted to say LONG_REQUEUE will take care of it but I'm open to reviewers' opinions.
There was a problem hiding this comment.
The previous MCP state is not available to this mapper, so the union-of-old-and-new approach is not practical without adding separate state. LONG_REQUEUE would eventually reconcile the MachineConfig—currently after one hour—but it leaves the ApprovedImage stale for up to that hour and is not a reliable event-driven cleanup mechanism.
I would avoid relying on it: on any MCP event, enqueue all cached MachineConfigs (or maintain the previous selector state separately), then let handle_machineconfig determine that previously selected objects are no longer live and delete their ApprovedImages.
There was a problem hiding this comment.
i.e.
.watches(mcps, Default::default(), move |_| {
let iter = mcp_ctx.mc_store.state().into_iter();
iter.filter_map(|mc| mc.metadata.name.as_deref().map(ObjectRef::new))
.collect::<Vec<_>>()
})hmm. still open to comments from reviewers
There was a problem hiding this comment.
Yes, that is the right fix. The mapper should enqueue every cached MachineConfig for any MachineConfigPool event, because handle_machineconfig will then evaluate each one against the current MCP selectors and delete images for MCs that are no longer selected. The current head still uses state_filter(|mc| is_live(&mcp, mc)), so the stale-image case remains.
.watches(mcps, Default::default(), move |_| {
mcp_ctx
.mc_store
.state()
.into_iter()
.filter_map(|mc| mc.metadata.name.as_deref().map(ObjectRef::new))
.collect::<Vec<_>>()
})This also handles selector widening and newly added MCPs; LONG_REQUEUE can remain only as a fallback for missed events.
| let url = mc.spec.os_image_url.as_ref(); | ||
| // Delete if moved out of scope or URL out of date | ||
| if !live || url.is_none_or(|url| *url != stored.spec.image) { | ||
| delete_approved_image(&stored, ctx.client.clone()).await?; |
There was a problem hiding this comment.
The assumption here is that each TEC-machine can have only a single approved-image (which may be true).
What happens if a CoreOS of a node is upgraded and the node fails to boot with the new (CoreOS) approved-image -- does it fallback to the previously working still-installed CoreOS version (or just keep trying and failing the new version) ?
There was a problem hiding this comment.
As discussed off-GitHub, we don't currently know enough about the upgrade process to be certain about when deletion is safe, and are currently leaving deletion open.
The MachineConfigPools are now additionally filtered by a label.
There was a problem hiding this comment.
I think we should also add the image from baseOSContainerImage, WDYT?
There was a problem hiding this comment.
Currently I can not see this documentation, but why prefer the base over the custom-image itself ?
Is it for not updating the ApprovedImage when switching between several images with the same base-image ?
The base-image and the custom-image can have different PCRs' values -- depending on the Containerfile modifications.
There was a problem hiding this comment.
In the current draft, both the ControllerConfig's image and the MachineConfigs' images would be added, and they'd be equal ApprovedImages
however
the ControllerConfig watch as drafted here does not work because it contains nullable required fields, which (like non-nullable required fields) Kopium turns into T (not Option<T>). Many real-world ControllerConfigs have some of these fields as null so the watch fails.
I'm considering adding a case to Kopium that changes this, but as we're not yet sure if such images will be configured in the ControllerConfig in the near future, I'm more strongly considering to leave out ControllerConfigs here for now. WDYT?
There was a problem hiding this comment.
moved ControllerConfig to a shelved branch, only watching MachineConfigs as originally for now
519fe93 to
3552022
Compare
15b51b8 to
f0a64c8
Compare
There was a problem hiding this comment.
Hey - I've found 8 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="api/v1alpha1/crds.go" line_range="38" />
<code_context>
// +kubebuilder:rbac:groups=trusted-execution-clusters.io,resources=trustedexecutionclusters/finalizers;machines/finalizers;attestationkeys/finalizers;approvedimages/finalizers,verbs=update
// +kubebuilder:rbac:groups=trusted-execution-clusters.io,resources=trustedexecutionclusters/status;machines/status;approvedimages/status;attestationkeys/status,verbs=get;patch;update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
+// +kubebuilder:rbac:groups=machineconfiguration.openshift.io,resources=machineconfigs;machineconfigpools,verbs=get;list;watch
// TrustedExecutionClusterSpec defines the desired state of TrustedExecutionCluster
</code_context>
<issue_to_address>
**OpenShift watches are forbidden**
When the OpenShift feature is installed using the repository’s committed RBAC artifacts without regenerating them, the new controller and reflectors list and watch cluster-scoped MachineConfigs and MachineConfigPools, but the committed RBAC artifacts do not include the permissions added by this marker. The API server returns Forbidden, so automatic ApprovedImage creation never works for installations using those artifacts.
Regenerate and commit the operator’s RBAC artifacts and production CSV permissions with the new MachineConfig and MachineConfigPool access.
</issue_to_address>
### Comment 2
<location path="operator/Cargo.toml" line_range="39" />
<code_context>
jsonwebtoken-openssl = "2.0.0"
+[features]
+openshift = ["trusted-cluster-operator-lib/openshift"]
+
[dev-dependencies]
</code_context>
<issue_to_address>
**Released operator omits image approval**
When the operator is built through the repository's standard release path, the normal build omits `--features openshift`, so the published operator excludes `openshift_images` and its MachineConfig watcher; users therefore get no automatic ApprovedImages from MachineConfigs.
Enable the `openshift` feature in the standard operator release build.
</issue_to_address>
### Comment 3
<location path="operator/src/openshift_images.rs" line_range="28" />
<code_context>
+ if mcp_labels.is_none_or(|ls| !ls.contains_key(MCP_APPROVED_IMAGE_LABEL)) {
+ return false;
+ }
+ let Some(mc_labels) = mc.metadata.labels.as_ref() else {
+ return false;
+ };
</code_context>
<issue_to_address>
**Unlabeled configs are not selected**
When a labeled MachineConfigPool selects an unlabeled MachineConfig using a valid `DoesNotExist` or `NotIn` expression, `is_live` returns false before evaluating the pool selector when `mc.metadata.labels` is absent, so reconciliation never approves the selected MachineConfig's OS image.
Evaluate the selector against an empty label set when the MachineConfig has no labels.
</issue_to_address>
### Comment 4
<location path="lib/src/reference_values.rs" line_range="22" />
<code_context>
+ if prefix.len() > 52 {
+ return Err(anyhow::anyhow!("prefix too long"));
+ }
+ let replaced = name.replace(['.', ':', '/', '@', '_'], "-");
+ let hash = hash(MessageDigest::sha1(), name.as_bytes())?;
+ let hashed = hex::encode(hash)[..10].to_string();
</code_context>
<issue_to_address>
**Uppercase image tags block approval**
When a selected MachineConfig's valid `osImageURL` contains uppercase characters, such as in its tag, `rfc1035` preserves uppercase characters in the generated name, and `add_mc_image` uses that value as the ApprovedImage metadata name; Kubernetes rejects the create, so the image is never approved.
Convert the generated resource name to lowercase before creating the ApprovedImage.
Also at `operator/src/openshift_images.rs:90`.
</issue_to_address>
### Comment 5
<location path="api/v1alpha1/crds.go" line_range="38" />
<code_context>
// +kubebuilder:rbac:groups=trusted-execution-clusters.io,resources=trustedexecutionclusters/finalizers;machines/finalizers;attestationkeys/finalizers;approvedimages/finalizers,verbs=update
// +kubebuilder:rbac:groups=trusted-execution-clusters.io,resources=trustedexecutionclusters/status;machines/status;approvedimages/status;attestationkeys/status,verbs=get;patch;update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
+// +kubebuilder:rbac:groups=machineconfiguration.openshift.io,resources=machineconfigs;machineconfigpools,verbs=get;list;watch
// TrustedExecutionClusterSpec defines the desired state of TrustedExecutionCluster
</code_context>
<issue_to_address>
**OpenShift read access is unconditional**
When the operator is installed on OpenShift with the `openshift` feature disabled, the RBAC generator includes cluster-wide `get`, `list`, and `watch` access to MachineConfigs and MachineConfigPools even when the operator is built without the `openshift` feature. On an OpenShift cluster, a compromised or misbehaving feature-disabled operator can still read those cluster configuration resources, exposing their contents.
Generate or distribute the MachineConfig permissions only for deployments built with the `openshift` feature.
</issue_to_address>
### Comment 6
<location path="operator/src/openshift_images.rs" line_range="91" />
<code_context>
+ };
+
+ let image_name = rfc1035(&format!("{mc_name}-{url}"), OSIMAGE_RESOURCE_PREFIX)?;
+ let labels = BTreeMap::from([(MC_LABEL.to_string(), mc_name.to_string())]);
+ add_approved_image(image_name, labels, url, ctx.client.clone()).await
+}
</code_context>
<issue_to_address>
**Long MachineConfig names block image creation**
When a selected MachineConfig has a valid name longer than 63 characters, `add_mc_image` copies the MachineConfig name into a Kubernetes label value, which is limited to 63 characters. A valid MachineConfig name longer than that makes the ApprovedImage create request fail validation, so the image is never approved.
Store the full MachineConfig name in an annotation or another field that permits longer values, or encode it into a valid bounded label value.
</issue_to_address>
### Comment 7
<location path="test_utils/src/lib.rs" line_range="1118" />
<code_context>
test_namespace: String,
manifests_dir: String,
test_name: String,
+ #[cfg(not(feature = "openshift"))]
delayed_approved_image: bool,
}
</code_context>
<issue_to_address>
**Test image fixtures are missing**
When integration tests run with the `openshift` test-utils feature enabled, `TestContext` skips applying the generated ApprovedImage manifests, but tests still query, delete, and verify those named fixtures; they fail because the resources are absent.
Keep applying the test ApprovedImage manifests when the OpenShift test-utils feature is enabled.
Also at `test_utils/src/lib.rs:1119`, `test_utils/src/lib.rs:1188`.
</issue_to_address>
### Comment 8
<location path="operator/Cargo.toml" line_range="39" />
<code_context>
jsonwebtoken-openssl = "2.0.0"
+[features]
+openshift = ["trusted-cluster-operator-lib/openshift"]
+
[dev-dependencies]
</code_context>
<issue_to_address>
**OpenShift test behavior is not inherited**
When tests build the operator with `--features openshift` without separately enabling the test-utils feature, enabling the operator’s `openshift` feature enables only the library feature, not `trusted-cluster-operator-test-utils/openshift`; test utilities therefore still apply ApprovedImage manifests and wait for them to be committed in builds that enable OpenShift only on the operator.
Wire the OpenShift test behavior to the same build feature, or expose and document a test feature that enables both crates’ OpenShift features.
Also at `test_utils/src/lib.rs:506`.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 8 findings to address first, and a labeled MachineConfigPool causes its selected MachineConfigs' OS image URLs to become ApprovedImages, so an unintended match can make an image trusted without a separate approval step. Those ApprovedImages are not removed automatically; reverting this change stops future creation but does not undo existing approvals.
Blocking findings: api/v1alpha1/crds.go:38, operator/Cargo.toml:39, operator/src/openshift_images.rs:28, lib/src/reference_values.rs:22, api/v1alpha1/crds.go:38, and 3 more
| #[cfg(feature = "openshift")] | ||
| pub const OSIMAGE_RESOURCE_PREFIX: &str = "osimage"; | ||
| #[cfg(feature = "openshift")] | ||
| pub const MCP_APPROVED_IMAGE_LABEL: &str = |
There was a problem hiding this comment.
It is worth having this value as an environment variable in case there is already an MCP that the administrator wants to be approved images
There was a problem hiding this comment.
admin can add labels, IMO that's overengineering
SpaceFace02
left a comment
There was a problem hiding this comment.
Looks good to me, Roy has a valid point regarding the cel's
for reuse Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
for reuse Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
for reuse Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
less indirection & can be used in non-anyhow contexts Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
instead of deriving Default, and then dropping it for unsupported structs, kopium has a flag do only include it for structs without multivalued enums which is enough for us. Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Gated behind "openshift" feature Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
5ff0fba to
67fc4c8
Compare
Insert osImageURLs of MachineConfigs as approved images. Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
for feature-gated MC detection. Move this to a separate section for separation of particular downstreams. Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
When enabled, do not create ApprovedImages. Drop an `image_ready` definition. Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Jakob-Naucke, SpaceFace02 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
fcc38a4
into
trusted-execution-clusters:main
Gated behind an
openshiftfeature, watch MachineConfigs and create and delete ApprovedImages based on their osImageURLs.With the same feature, do not create ApprovedImages in tests.
Happy to resume this work, initial commit is over a year old 😅
Summary by Sourcery
Add feature-gated OpenShift support for automatically creating ApprovedImages from MachineConfig OS image URLs while keeping non-OpenShift behavior unchanged.
New Features:
Bug Fixes:
Enhancements:
Build:
CI:
Documentation:
Tests:
Chores: