Configurable automatic retries for failed actions (policy, backoff, rollout and auto-assignment support)

[laverman]

Summary

Introduce first-class retry policies for failed deployment actions, including delay/backoff behavior, status code matching rules, and support in rollouts plus auto-assignments.

Goal: Improve success rates for transient failures while keeping resource usage controlled and observable.

Problem Statement

Device-side installations can fail for transient reasons (for example unstable connectivity, temporary service issues).
In many cases, a retry succeeds, but today recovery is mostly manual.
This causes operational overhead and lower rollout efficiency.

Scope

1. Add retry policy configuration at action level.
2. Extend rollout configuration to propagate retry policy into created actions.
3. Extend auto-assignment configuration to propagate retry policy into created actions.
4. Execute retries automatically on eligible failures.
5. Persist retry metadata for analytics and filtering.

Proposal

1. Retry Policy on Actions

When creating an action, allow configuring a retry policy with:

• maxRetries: number of retry attempts after the initial failure.
• retryDelayPolicy: delay strategy between attempts.
• statusCodeMatcher: defines which error codes are retry-eligible.

Suggested request shape:

{
  "retry": {
    "maxRetries": 3,
    "delayPolicy": {
      "type": "exponential",
      "initialDelaySeconds": 30,
      "maxDelaySeconds": 900,
      "multiplier": 2.0,
      "jitter": "full"
    },
    "statusCodeMatcher": {
      "mode": "allowlist",
      "codes": [200, 503, 901],
      "ranges": [[500, 599]]
    }
  }
}

2. Triggering Behavior

If device feedback closes with error, for example:

{
  "status": {
    "execution": "closed",
    "result": {
      "finished": "error"
    },
    "code": 503
  }
}

backend evaluates retry policy:

• retries remaining?
• status code eligible?
• if yes: schedule next attempt using delay policy, increment retryCount, keep same action ID, keep/return action to running
• if no: finalize as terminal failed

3. Delay/Backoff

Support at least:

• fixed delay
• exponential backoff
• optional jitter to avoid retry bursts/thundering herd

Design constraints:

• enforce sane caps (min/max delay, max retries)
• deterministic and auditable next-run time

4. Status Code Matching Model

Open design question to settle in this ticket:

• allowlist only
• denylist only
• both
• code ranges support

Recommendation:

• support both allowlist and denylist modes
• support single codes plus ranges
• define precedence clearly

Suggested semantics:

• mode = allowlist: retry only when code matches configured codes/ranges
• mode = denylist: retry for all error codes except configured codes/ranges
• if matcher omitted: retry all error codes (subject to maxRetries)

5. Rollouts and Auto-Assignments

Rollout definitions and auto-assignment definitions should support retry policy fields so each generated action gets per-action retry configuration.

Expected behavior:

• rollout-level retry policy acts as default for generated actions
• auto-assignment-level retry policy acts as default for generated actions
• persisted action always contains resolved effective policy for auditability

Data Model Changes (Action)

Add/extend fields:

• retry.maxRetries
• retry.retryCount
• retry.delayPolicy
• retry.statusCodeMatcher
• retry.nextRetryAt
• retry.lastRetryStatusCode
• retry.history (attempt number, timestamp, status code, computed delay, outcome)

Why same action ID by default

Pros:

• preserves full lifecycle in one object
• avoids duplicate action explosion
• avoids repeated user confirmation flows
• easier operator comprehension and filtering

Cons:

• requires careful handling of close-error to running transitions
• existing assumptions that closed is terminal may need updates

Alternative (new action per retry) remains possible but is not preferred for initial implementation due to confirmation and traceability complexity.

API and UX Considerations

• expose retry policy in action, rollout, and auto-assignment APIs
• expose effective resolved action retry policy
• add filters:
  • retryCount greater than N
  • hadRetries true/false
  • retry-triggering status code
• expose metrics:
  • retries attempted
  • retries succeeded
  • retries exhausted
  • top retry-triggering status codes

Acceptance Criteria

• Action API supports retry policy including delay/backoff and status code matcher
• Rollout API supports retry policy defaults for generated actions
• Auto-assignment API supports retry policy defaults for generated actions
• Backend retries eligible failures automatically with configured delay strategy
• Same action ID is reused across retries
• retryCount and retry history are persisted and queryable
• Status code matching supports codes and ranges per selected matcher mode
• Action becomes terminal failed when retries are exhausted or ineligible
• Filtering and metrics support operational analysis

Open Questions

1. Which matcher modes should be in v1: allowlist only, denylist only, or both?
2. If both modes are supported, do we need explicit precedence rules beyond mode exclusivity?
3. Should default matcher be retry-all-errors or retry-none unless configured?
4. Should backoff policy be immutable after action creation?
5. Do we need rollout and auto-assignment policy inheritance override flags?

[strailov]

Thanks for the detailed proposal. A few concerns before we commit to this scope:

• Complexity

  • This is a large, cross-cutting feature — it touches the action model, rollouts, auto-assignments, the DDI feedback path, plus new filtering and metrics.
  • That's a significant surface to design, implement, and maintain.

• Data model / migration cost

  • The action table is one of the largest tables in most hawkBit deployments.
  • Adding retry columns (and especially a retry.history structure) means schema migrations that can be heavy on big installations.
  • Persisting per-attempt history risks noticeable table growth over time.

• Existing capability

  • There's already a lightweight mechanism for this: the device controls when it reports status.
  • It can simply retry its installation locally and only report a terminal result once it has genuinely exhausted its own attempts.
  • Transient connectivity/service failures are often best handled closest to the device.

• Proposed simpler alternative

  • Rather than a full server-side retry engine, we could expose a REST endpoint to retry an action on the hawkBit side. This would:
    • reuse the existing action lifecycle instead of introducing new close-error → running transitions,
    • require small, contained changes (no heavy migrations, no per-attempt history table),
    • leave backoff/scheduling policy to the caller (device or an external orchestrator), which keeps hawkBit's responsibilities focused.

The automatic policy/backoff/matcher engine could still be layered on later if there's proven demand, but starting with device-side retry + a retry endpoint gives most of the operational value at a fraction of the cost and risk.

Would this direction be acceptable as a first step?

[laverman]

@strailov Thanks for your feedback. I agree, let's make one step at a time and start with the manual retry by offering the REST endpoint on the management API first.