Add credential refresh for track2 SDKs

[Rajdeep Chauhan (rajdeepc2792)]

Refresh Token for track2 sdk BearerTokenPolicy flow per client:

flowchart LR
    A["BearerTokenPolicy cache<br/>per client pipeline"] -->|"defeated by<br/>ExpiresOn = zero"| B["our GetToken called<br/>on every request"]
    B --> C["azidentity / MSAL cache<br/>per credential"]
    C -->|".....same JWT until..."| D["Rebuild() throws<br/>credential away"]
    D -->|"→ new JWT"| C

Loading

Testing Approach

• During local cluster creation, removed the mockFPSP privileges on the managed resource group just before it attempted to retry for a Track2 SDK API.
• This attempted multiple retries for 10mins, for each call the Trace must show a new AccessToken with new uti.
• The above testing is done and confirmed on local.

Which issue this PR addresses:

Fixes https://issues.redhat.com/browse/ARO-30032

What this PR does / why we need it:

• The RP built one FP certificate credential per cluster manager and shared it across every track2 SDK client, so a single access token was presented for the entire install.
• ARM frontdoor caches a principal's role assignments per access token, which pinned us to the RBAC snapshot cached when that token was first seen
• role assignments created later were invisible, and retries re-presented the same token to the same frontdoor instance and failed identically.
• Cluster installs in Fairfax failed consistently at managed resource group creation for ~40 minutes.
• This makes the FP credentials refreshable: refreshable.NewFPTokenCredential wraps the credential so Rebuild() discards it and its token cache, and fpCredRefresher rebuilds the track1 authorizer plus both track2 FP credentials together.
• AuthorizationRetryingAction now takes a refreshable.Rebuilder and drives that refresher, so each auth retry presents a genuinely new token and ARM re-evaluates current role assignments.
• ensureResourceGroup is promoted from steps.Action to AuthorizationRetryingAction.

Test plan for issue:

• Unit tests added in pkg/util/refreshable covering credential rebuild (new token per rebuild), rebuild failure leaving the previous credential usable, and NewMultiRebuilder ordering/short-circuit on error.
• Existing pkg/util/steps and pkg/cluster suites pass; pkg/cluster/adminupdate_test.go updated for the ensureResourceGroup step type change.
• End-to-end behavior depends on ARM frontdoor caching and cannot be reproduced in unit tests. Manual verification: create a cluster in an affected Fairfax region and confirm managed resource group creation no longer stalls, and that retry attempts carry distinct bearer tokens.

Is there any documentation that needs to be updated for this PR?

No — internal credential-handling fix with no API, config, or operator-facing surface change.

How do you know this will function as expected in production?

The retry path is unchanged apart from the credential rebuild, and the existing AuthorizationRetryingAction logging already records each auth retry; a rebuild failure is logged and surfaced as the step error rather than being swallowed.
A failed Rebuild() leaves the prior credential in place, so the worst case is the current behavior rather than a broken client.

[Copilot]

Copilot review overview

🟡 Changes recommended

Track 2 client pipelines retain their cached access tokens after the wrapped credential is rebuilt, preventing the intended authorization refresh.

Review effort: Balanced
Findings: 1

Open (1)

• Rebuilding credential does not invalidate BearerTokenPolicy token cache · New

What changed in this PR

Adds coordinated credential rebuilding for Track 1 and Track 2 Azure SDK clients during authorization retries.

Changes:

• Introduces refreshable token credentials and multi-credential rebuilding.
• Wires cluster authorization retries to refresh all FP credentials.
• Adds credential rebuild tests and updates step expectations.

File	Description
pkg/​util/​steps/​refreshing.go	Generalizes retries to rebuild credentials.
pkg/​util/​refreshable/​refreshable.go	Adds refreshable Track 2 credentials.
pkg/​util/​refreshable/​refreshable_test.go	Tests rebuild behavior.
pkg/​cluster/​install.go	Uses coordinated credential refreshing.
pkg/​cluster/​cluster.go	Constructs and wires refreshable credentials.
pkg/​cluster/​adminupdate_test.go	Updates expected step type.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟢 Approval recommended

The credential refresh path is consistently integrated and covered by focused failure and ordering tests.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Rebuilding credential does not invalidate BearerTokenPolicy token cache

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation consistently refreshes both SDK generations and includes focused coverage for token replacement and failure behavior.

Review effort: Balanced
Findings: None

[Tanmay Satam (tsatam)]

LGTM. One optional suggestion to split the refreshable file to isolate track1 and track2 refreshable implementations from each other.

[Copilot]

🔵 Needs a closer look

The implementation changes production authentication and token-caching behavior across many cluster operations, warranting final human review.

0 open findings

🧠 Review effort: Balanced

[Copilot]

🟢 Approval recommended

The bearer-token cache issue is addressed with focused tests covering successful and failed rebuild paths.

0 open findings

🧠 Review effort: Balanced