benchmark-list: expose Go runtime metrics to disambiguate memory regressions

[Gaurav598]

What type of PR is this?

/kind bug

What this PR does / why we need it:
This change exposes additional Go runtime metrics for the benchmark-list benchmark using the existing GenericPrometheusQuery infrastructure.

The benchmark currently primarily surfaces container_memory_working_set_bytes, which can increase when allocation rate decreases and Go GC runs less frequently. This can make allocation improvements appear as memory regressions even when the live heap remains unchanged.

This PR adds GenericPrometheusQuery measurements for:

• go_memstats_heap_inuse_bytes
• rate(go_gc_duration_seconds_count[%v])
• rate(go_memstats_alloc_bytes_total[%v])
  and registers these metrics in Perfdash so they are displayed alongside the existing benchmark results.

Files changed:

• clusterloader2/testing/list/modules/measurements.yaml – add GenericPrometheusQuery measurements.
• perfdash/config.go – register GenericMeasurements for the benchmark-list dashboard.
• perfdash/parser_test.go – add coverage for GenericPrometheusQuery parsing.

The implementation is additive and does not modify the existing ResourceUsageSummary pipeline.

Which issue(s) this PR fixes:
Fixes #4140

Special notes for your reviewer:
This change intentionally uses the existing GenericPrometheusQuery pipeline instead of extending ResourceUsageSummary.
Relevant tests executed:

• go test ./... (perfdash)
• go test ./pkg/measurement/common (clusterloader2)

The implementation only adds new benchmark metrics and Perfdash registration without changing existing benchmark behavior.

[kubernetes-prow]

Hi @Gaurav598. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Gaurav598
Once this PR has been reviewed and has the lgtm label, please assign wojtek-t for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• clusterloader2/testing/list/OWNERS
• perfdash/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Jefftree]

Please test this with a local perfdash instance.

[Gaurav598]

@Jefftree
I have validated the Perfdash configuration locally.
To avoid fabricating synthetic data, I pointed a local Perfdash instance at the live public GCS artifacts for ci-kubernetes-benchmark-list using a temporary local config.

As shown in the attached screenshot, the GenericMeasurements category is discovered correctly and Perfdash successfully parses and renders the existing GenericPrometheusQuery artifact (kube-apiserver memory usage) for the benchmark-list job.
The new heap in-use, GC rate, and allocation rate metrics introduced by this PR are not present in the current public artifacts because they will only be produced after benchmark-list runs with this change.

This confirms that the Perfdash parser/configuration wiring added in this PR is working as expected. Please let me know if you'd like me to validate anything else.

[Jefftree]

/ok-to-test

[Jefftree]

/test pull-perf-tests-benchmark-list-proto

[Gaurav598]

Hi @Jefftree, just a gentle ping on this PR.
I have updated the PromQL selectors based on the TSDB snapshot and removed the synthetic test as requested. The CI is fully green. Please let me know if everything looks good or if there is anything else you need from me!

[Jefftree]

Task looks good. I'm still trying to reason if this gives us the signal we want, sorry will need a bit more time thinking about it. I had thought these metrics would give us a clear diff but it doesn't seem like it.

[Gaurav598]

Hi @Jefftree, no worries at all, take your time! I'm glad the implementation itself looks good.
If you come to the conclusion that different metrics (perhaps go_memstats_heap_alloc_bytes or specific percentiles) might provide a clearer signal to distinguish between benign GC shifts and actual memory regressions, please let me know.
I'm happy to update the queries or explore other options whenever you are ready.

[Gaurav598]

Hi @Jefftree, just following up on this PR. It’s been a couple of weeks since the last update, so I wanted to check if you’ve had a chance to reach a conclusion on whether these metrics provide the signal we’re looking for.

If you’d prefer a different approach or different metrics, I’m happy to make the necessary changes. Thanks!