fix: bound the systemctl retry budget in cse_helpers.sh

[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

What

systemctlEnableAndStart / systemctlEnableAndStartNoBlock hardcode systemctl_restart 100 5 $timeout $service.

The retry count is not a time bound, and the 30 / 300 a caller passes is only the per-systemctl-call timeout, not a retry budget. systemctlEnableAndStart localdns 30 reads like "give up after 30s"; it can actually run for over an hour.

I measured this with a fast-failing unit:

scenario	before	after
critical unit fails fast	498s (rc=1)	122s (rc=2, 24 attempts)
optional unit fails fast	498s	10s

DefaultCSETimeout is 900s (pkg/agent/datamodel/const.go). A single bad unit eating 498s of that means downstream provisioning steps never run, and their specific error codes get replaced by a generic CSE timeout — which is strictly worse for diagnosis than failing fast.

This isn't hypothetical. cse_config.sh::configureLocalDNSExporterSocket already carries a per-caller workaround for exactly this problem:

# systemctlEnableAndStartNoBlock would hit its retry loop (~100 retries × 5s) for a missing unit.

The 100 5 values date to #7339 and were never deliberately tuned (#8105 only moved the status logging).

Changes

_systemctl_retry_svc_operation

• Optional maxBudget (7th arg, default 0 = disabled) — the same convention _retry_file_curl_internal already uses in this file. Caps the per-attempt timeout to the remaining budget and returns 2 when exhausted.
• Adds the check_cse_timeout guard that every other retry helper here already has, so we exit with a specific code instead of running the CSE window to zero.
• Bounds the journal dump: journalctl -u $svc --no-pager -n 50. It previously ran unbounded on every failed attempt, so log volume grew quadratically with the retry count.

systemctlEnableAndStart / NoBlock

• Optional 3rd retries param; applies a 120s budget by default.
• systemctl enable retries 120 → 3. It only writes symlinks on disk and runs immediately after a successful restart, so a 600s tail cannot help.

Callers — a small retry count for units whose failure is already explicitly non-fatal: mig-partition, node-exporter(+.path), nvidia-dcgm(+-exporter), localdns-exporter.socket, aks-localdns-hosts-setup.timer, aks-log-collector.timer, measure-tls-bootstrapping-latency. There are enough of these that a full 120s budget each would exceed the CSE window on its own.

Worth calling out: ensureMigPartition is documented as "this is expected to fail and work only on next reboot" — and is currently retried for ~500s.

containerd, kubelet, localdns, and nvidia-gridd keep the default budget.

Compatibility

Every new parameter is optional and defaults to today's values, so old and new copies of these scripts interoperate in both directions. systemctl_stop / systemctl_disable are unchanged (budget defaults to 0). cse_helpers.sh and cse_config.sh ship in the same CustomData blob, so the shared constants are versioned together.

Testing

• make shellspec (docker): 939 examples, 0 failures
• 9 new examples covering the retry loop, which previously had no direct coverage — every existing caller mocked the helper. Covers: budget trips before retry exhaustion, legacy behavior when no budget is passed, success path, bounded journalctl, arg passthrough, and caller-supplied retry counts.
• make validate-shell: shellcheck findings byte-identical to main (the gate has pre-existing failures in unrelated files).
• go test ./pkg/agent/...: ok
• Wall-clock numbers in the table above measured in the shellspec container.

No testdata regeneration needed — cse_helpers.sh is not embedded in the parser snapshots.

[github-actions]

Windows Unit Test Results

  3 files   13 suites   50s ⏱️
404 tests 404 ✅ 0 💤 0 ❌
407 runs  407 ✅ 0 💤 0 ❌

Results for commit 3b9a7d6.

[Copilot]

Pull request overview

Bounds systemctl retries during Linux VHD builds and node provisioning.

Changes:

• Adds operation budgets and bounded journal output.
• Reduces retries for optional services.
• Adds ShellSpec coverage for retry behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
cse_helpers.sh	Adds retry budgets and tuning constants.
cse_config.sh	Applies reduced retries to optional services.
cse_main.sh	Limits log collector timer retries.
cse_retry_helpers_spec.sh	Tests budgets, passthrough, and journal limits.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.