clusterloader2: use precise timestamps for ResourceClaim latency

[Gaurav598]

What this PR does

Uses informer receive timestamps for ResourceClaim and Pod creation measurements in resourceclaim_allocation_latency.go to preserve sub-second precision.

Why

metadata.creationTimestamp has second-level precision, which can make short ResourceClaim lifecycle measurements inaccurate.

This affects measurements such as:

• claim_allocation
• pod_create_to_claim_create

The allocation path already uses the informer receive time, so this change aligns the ResourceClaim creation and Pod creation paths with the same approach.

Changes

• Inject a clock.Clock into the ResourceClaim allocation latency measurement.
• Capture the informer receive time when ResourceClaims are observed.
• Use the captured receive time for createPhase.
• Capture the informer receive time when Pods are observed.
• Use the cached Pod receive time for podCreatePhase.
• Keep the fetchPodCreateTime fallback based on CreationTimestamp unchanged.

Tests

Added regression tests using FakeClock covering:

• sub-second ResourceClaim allocation latency
• sub-second Pod-to-ResourceClaim creation latency
• already-allocated ResourceClaims being ignored

Tests run:

go test ./pkg/measurement/common/slos/... -v -run 'TestResourceClaim'
go test ./pkg/measurement/common/slos/... -count=1

Both pass.

Fixes #3889

[kubernetes-prow]

Hi @Gaurav598. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Gaurav598
Once this PR has been reviewed and has the lgtm label, please assign wojtek-t for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• clusterloader2/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[isaac-dasan]

I ran into this same issue and had written an equivalent patch before I saw this PR. I'd commented on #3889 back on Aug 30 asking to take it but never actually opened anything. No point filing a duplicate, so here's some validation data instead. The two implementations are functionally identical, so all of this applies directly to your branch.

Verification against this PR's head (4f57b14)

I took the FakeClock regression tests I'd written separately and dropped them onto your branch. Both pass:

• sub-second latency when every object shares the same creationTimestamp second
• already-allocated claims getting ignored on first observation

go build ./... and go vet ./pkg/measurement/common/slos/... are both clean.

Before and after on a real cluster

kind v1.36.4, 3 nodes, dra-example-driver, 8 GPU-claiming Job pods. Same config, same cluster, only the binary changes (master 4f4d01ea3 vs this PR):

Metric	before	after
claim_allocation P50	636 ms	84 ms
claim_allocation P90	1110 ms	111 ms
pod_create_to_claim_create P50	0 ms	15.6 ms
pod_create_to_claim_create P90	0 ms	35.2 ms

pod_create_to_claim_create reproduced the exact all-zero symptom from #3889.

The per-claim claim_allocation numbers on master show it best: 186, 314, 522, 636, 770, 868, 953, 1110 ms. That's eight samples spread almost uniformly across [0, 1.1s], for pods whose real latency was between 63 and 111 ms. Classic measured = true + frac(create_time) with frac ~ Uniform[0, 1s). With this change those same eight come out as 63, 74, 76, 84, 90, 94, 99, 111 ms.

PodGetCalls was 0 on both runs, so the fetchPodCreateTime fallback that this PR leaves on CreationTimestamp never actually gets hit. Agree it's fine to leave alone.

Two notes, neither blocking

1. pod_create_to_claim_create now subtracts stamps coming from two different informers, so what it really measures is true + (claim_watch_lag - pod_watch_lag). addPodEvent runs inline in the handler while claims go through a workqueue, so the pod stamps will drift late under churn.
2. Together with the clamp at phase_latency.go:143 (negative goes to 0), that biases the metric downward at high load instead of surfacing the skew.

Neither one touches claim_allocation, which is the SLO metric and pulls both endpoints from the same informer. Might still be worth a comment saying the metric is really controller reaction plus inter-watch skew, just so nobody over-reads it later.

LGTM from me, though I'm not an org member so that doesn't carry any Prow weight.

/cc @justinsb @alaypatel07 - this needs /ok-to-test from a member before CI will run.

[Gaurav598]

Thanks @isaac-dasan for the independent validation and the real-cluster before/after data. This is very helpful, especially the reproduction of the all-zero pod_create_to_claim_create behavior and the improvement in claim_allocation measurements.

I agree with the two notes around cross-informer timestamp skew and the existing clamp behavior. I'll keep them in mind as follow-up considerations; I don't think they block this fix.

Thanks again for the thorough validation!