flagger_canary_weight metric is missing the name label

[dkulchinsky]

Describe the bug

the flagger_canary_weight metric is missing the name label, indicating the Canary's name.

for example:

flagger_canary_weight{app_kubernetes_io_instance="flagger", app_kubernetes_io_name="flagger", app_kubernetes_io_version="1.44.0", instance="172.26.3.33:8080", job="kubernetes-pods", kubernetes_namespace="flagger-system", kubernetes_pod_name="flagger-5b546d7775-xrvh8", namespace="my-app", node="gke-sbx-usc1-platform-9b94eeb5-csxz", node_name="gke-sbx-usc1-platform-9b94eeb5-csxz", pod_template_hash="5b546d7775", workload="podinfo-istio-ambient"} 0

flagger_canary_weight{app_kubernetes_io_instance="flagger", app_kubernetes_io_name="flagger", app_kubernetes_io_version="1.44.0", instance="172.26.3.33:8080", job="kubernetes-pods", kubernetes_namespace="flagger-system", kubernetes_pod_name="flagger-5b546d7775-xrvh8", namespace="my-app", node="gke-sbx-usc1-platform-9b94eeb5-csxz", node_name="gke-sbx-usc1-platform-9b94eeb5-csxz", pod_template_hash="5b546d7775", workload="podinfo-istio-ambient-primary"} 100

To Reproduce

run flagger, create some canaries, shift traffic weights and scrape it with prometheus.

Expected behavior

the flagger_canary_weight metric should have the name label indicating the canary's name.

Additional context

• Flagger version: 1.44.0
• Kubernetes version: 1.34.8
• Service Mesh provider: gatewayapi
• Ingress provider: N/A

[dkulchinsky]

I'm working on fix PR for above, but while testing things I noticed another odd issue.

flagger emits metrics for the following 3 canaries:

flagger_canary_status{app_kubernetes_io_instance="flagger", app_kubernetes_io_name="flagger", app_kubernetes_io_version="1.44.0", instance="172.26.3.33:8080", job="kubernetes-pods", kubernetes_namespace="flagger-system", kubernetes_pod_name="flagger-5b546d7775-xrvh8", name="httpbin-istio-ambient", namespace="my-app", node="gke-sbx-usc1-platform-9b94eeb5-csxz", node_name="gke-sbx-usc1-platform-9b94eeb5-csxz", pod_template_hash="5b546d7775"} 1

flagger_canary_status{app_kubernetes_io_instance="flagger", app_kubernetes_io_name="flagger", app_kubernetes_io_version="1.44.0", instance="172.26.3.33:8080", job="kubernetes-pods", kubernetes_namespace="flagger-system", kubernetes_pod_name="flagger-5b546d7775-xrvh8", name="podinfo-flagger", namespace="flagger-system", node="gke-sbx-usc1-platform-9b94eeb5-csxz", node_name="gke-sbx-usc1-platform-9b94eeb5-csxz", pod_template_hash="5b546d7775"} 1

flagger_canary_status{app_kubernetes_io_instance="flagger", app_kubernetes_io_name="flagger", app_kubernetes_io_version="1.44.0", instance="172.26.3.33:8080", job="kubernetes-pods", kubernetes_namespace="flagger-system", kubernetes_pod_name="flagger-5b546d7775-xrvh8", name="podinfo-istio-ambient", namespace="my-app", node="gke-sbx-usc1-platform-9b94eeb5-csxz", node_name="gke-sbx-usc1-platform-9b94eeb5-csxz", pod_template_hash="5b546d7775"} 1

however, the cluster only has 2 canaries:

❯ k get canary -A
NAMESPACE   NAME             STATUS      WEIGHT   LASTTRANSITIONTIME
my-app      httpbin-canary   Succeeded   0        2026-07-22T13:53:10Z
my-app      podinfo-canary   Succeeded   0        2026-07-22T14:27:10Z

perhaps there's a missing step to remove deleted canaries from metrics registry?

EDIT: confirmed, the controller's DeleteFunc only removes the entry from an in-memory map:

flagger/pkg/controller/controller.go

Lines 178 to 184 in 7a8f58b

	DeleteFunc: func(old interface{}) {
	r, ok := checkCustomResourceType(old, logger)
	if ok {
	ctrl.logger.Infof("Deleting %s.%s from cache", r.Name, r.Namespace)
	ctrl.canaries.Delete(fmt.Sprintf("%s.%s", r.Name, r.Namespace))
	}
	},

it never touches the Prometheus registry. So every per-canary series (flagger_canary_status, flagger_canary_weight, flagger_canary_duration_seconds, flagger_canary_metric_analysis, flagger_canary_successes_total, flagger_canary_failures_total) is retained for the process lifetime.

[dkulchinsky]

@aryan9600 @stefanprodan fix for above two issues for your consideration: #1961

It would be wonderful if this can make it into 1.45 🙏🏼

[dkulchinsky]

After further review, I realized name in all metrics (except _wieght where it's missing) is actually spec.targetRef.name and not the Canary resource name, this threw me off at first because we tend to name the Canaries the same as the target.

as such, I've changed my proposal in my PR to the following:

1. for consistency remove name from flagger_canary_weight (i.e. revert to original state)
2. add a new label - canary which would be the Canary resource name to all the metrics

We end up with the following set of labels per metric:

Metric	Labels (after)
flagger_canary_status	canary, name, namespace
flagger_canary_duration_seconds	canary, name, namespace
flagger_canary_metric_analysis	canary, name, namespace, metric
flagger_canary_weight	canary, workload, namespace
flagger_canary_successes_total	canary, name, namespace, deployment_strategy, analysis_status
flagger_canary_failures_total	canary, name, namespace, deployment_strategy, analysis_status