Skip to content

perflens: re-base run timestamps so runs share one axis - #4267

Open
Jefftree wants to merge 4 commits into
kubernetes:masterfrom
Jefftree:perflens-normalize-time
Open

Jefftree wants to merge 4 commits into
kubernetes:masterfrom
Jefftree:perflens-normalize-time

Conversation

@Jefftree

@Jefftree Jefftree commented Aug 14, 2026 •

Copy link
Copy Markdown
Member
  • Ingest re-bases each run's samples to a common anchor, so one time range works for every run and selecting two runs overlays them instead of showing two spikes days apart. perflens_run_start_timestamp_seconds keeps the original start.
  • Adds a Control Plane dashboard: process resources, Go runtime, etcd disk.
  • Turns off the Grafana news feed and analytics calls.

Basically allows us to loosely compare two runs in the same dashboard and we don't need to hunt for the time as we align every run to start at 2000/1/1 0:00 UTC.

image image

@kubernetes-prow kubernetes-prow Bot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 14, 2026
@kubernetes-prow kubernetes-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 14, 2026
@Jefftree
Jefftree force-pushed the perflens-normalize-time branch from 07cea7d to a268d40 Compare August 14, 2026 17:12
@Jefftree

Copy link
Copy Markdown
Member Author

/assign @serathius

okay let's iterate this.

Comment thread perflens/README.md Outdated
deliberate: Grafana has no duration axis mode, so an axis labelled `00:00`, `00:30`,
`01:00` reads as elapsed time for free.

The offset is computed once per run, from the earliest `minTime` across its Prometheus

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Can we compute minTime based on Cluster loader steps?

environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_NEWS_NEWS_FEED_ENABLED=false

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this?


const twoHoursMS = int64(2 * 60 * 60 * 1000)

// runSpanStepMS is the cadence used for values that hold for a whole run, such as

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I have high requirements for vibecoded project, but I cannot understand this comment at all.

@serathius

Copy link
Copy Markdown
Member

Overall I like the direction of automatically aligning runs on timeline, however I'm worried that rewriting prometheus snapshots adds complexity and performance implication. I used thanos to avoid rewriting blocks for exactly this reason.

My ask is whether we can make this change semi reviewable so we can discuss prometheus setup and code architecture. Would splitting this PR into steps would be useful or we should merge without code review?

@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jefftree
Once this PR has been reviewed and has the lgtm label, please ask for approval from serathius. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Jefftree
Jefftree force-pushed the perflens-normalize-time branch from 6aa2186 to 25b1b82 Compare August 18, 2026 14:49
@kubernetes-prow

Copy link
Copy Markdown
Contributor

@Jefftree: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-perf-tests-verify-lint 25b1b82 link true /test pull-perf-tests-verify-lint

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants