Skip to content

Delete stale primary HPAs when autoscalerRef is removed - #1965

Open
nedal87 wants to merge 1 commit into
fluxcd:mainfrom
nedal87:fix/stale-primary-hpa-cleanup
Open

Delete stale primary HPAs when autoscalerRef is removed#1965
nedal87 wants to merge 1 commit into
fluxcd:mainfrom
nedal87:fix/stale-primary-hpa-cleanup

Conversation

@nedal87

@nedal87 nedal87 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Problem

Removing spec.autoscalerRef from a Canary leaves the generated primary HPA
behind. It continues scaling the primary Deployment after autoscaling was
intended to be disabled.

Fixes #1654. Related: #259.

Solution

When autoscalerRef is absent, Flagger searches the Canary namespace for a
stale primary HPA and deletes it only when all of these checks pass:

  • the Canary has a UID;
  • the HPA name ends in -primary;
  • it targets <targetRef.name>-primary;
  • its controller owner reference matches the exact Canary UID.

No HPA is deleted based on its name alone. This addresses the ownership-safety
concern raised in #1030.

Successful cleanup is memoized per Canary UID, so Flagger performs one
namespace-scoped HPA list after startup or after autoscalerRef is removed,
rather than on every analysis tick. Failures are reported and retried without
blocking normal Canary reconciliation.

A pure Canary UpdateFunc cleanup would miss references removed while Flagger
was unavailable and HPAs that were already stale before upgrading.

No RBAC changes are required; the upstream chart and kustomize manifests
already grant Flagger HPA list and delete permissions. This change is limited
to autoscaling/v2 HorizontalPodAutoscalers.

Testing

Added tests for:

  • cleanup after removing autoscalerRef;
  • cleanup of an already-stale HPA after controller startup;
  • preservation of unmanaged and differently owned HPAs;
  • LIST and DELETE failure retries;
  • disable, re-enable, and disable lifecycle;
  • primary workload health.

Validated with:

  • go test ./...
  • race tests for pkg/canary and pkg/controller
  • go vet and go build ./...
  • live kind tests covering cleanup, re-enablement, ownership safety, and
    Canary deletion garbage collection

Delete Flagger-managed primary HPAs after autoscalerRef is removed.

Require the Canary controller owner and expected primary target.

Keep discovery failures from blocking Canary reconciliation.

Signed-off-by: Nedal Eskaf <nedaleskaif87@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Primary HPA does not get removed automatically after autoscalerRef is removed from Canaray

1 participant