Add transport URL secret rotation with consumer finalizer - #982
Conversation
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages. Design: 1. Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile. 2. Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs. 3. Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status. 4. Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness. 5. Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret. The same pattern applies to notification transport URL secrets and application credential secrets where applicable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lmiccini The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Build failed (check pipeline). Post ✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 3h 01m 44s |
|
/test telemetry-operator-build-deploy |
|
recheck |
| if instance.Status.NotificationsURLSecret != nil { | ||
| rotationPending = *instance.Status.NotificationsURLSecret != "" && | ||
| *instance.Status.NotificationsURLSecret != currentNotifSecret |
There was a problem hiding this comment.
The instance.Status.NotificationsURLSecret variable is unconditionally overwritten before rotationPending is calculated and it never changes so that means that here, the "*instance.Status.NotificationsURLSecret != currentNotifSecret" condition is always false, and thus this controller never activates the 60-second grace period.
This also happens in ceilometer_controller, but it is correctly handled in the cloudkitty_controller.
This might do the trick:
| if instance.Status.NotificationsURLSecret != nil { | |
| rotationPending = *instance.Status.NotificationsURLSecret != "" && | |
| *instance.Status.NotificationsURLSecret != currentNotifSecret | |
| rotationPending := oldNotifSecret != "" && oldNotifSecret != currentNotifSecret |
| rotationPending := false | ||
| if instance.Status.NotificationsURLSecret != nil { | ||
| rotationPending = *instance.Status.NotificationsURLSecret != "" && | ||
| *instance.Status.NotificationsURLSecret != currentNotifSecret | ||
| } |
There was a problem hiding this comment.
Same as in autoscaling, this could be:
| rotationPending := false | |
| if instance.Status.NotificationsURLSecret != nil { | |
| rotationPending = *instance.Status.NotificationsURLSecret != "" && | |
| *instance.Status.NotificationsURLSecret != currentNotifSecret | |
| } | |
| rotationPending := oldNotifSecret != "" && oldNotifSecret != currentNotifSecret |
2 participants
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages.
Design:
Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile.
Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs.
Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status.
Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness.
Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret.
The same pattern applies to notification transport URL secrets and application credential secrets where applicable.