feat(seinode): maintenance hold for nodeConfig nodes (spec 010, PLT-1390)

[bdchatham]

Summary

This PR implements spec 010 (#590). It stacks on #594 (spec 009), because it extends that PR's start guard.

spec.maintenance.hold parks seid at the sidecar start gate. The pod stays alive and the data volume stays mounted, so an operator can run seid rollback and chain-halt diagnostics through kubectl exec -c seid. status.maintenanceHold records the hold in effect. Every plan that changes it ends with record-maintenance-hold.

Requested → in effect	Plan
Immediate ← none / AfterExit	mark-not-ready → stop-seid → record(Immediate)
AfterExit ← none	mark-not-ready → record(AfterExit): seid runs until it exits, then parks
AfterExit ← Immediate (start once)	observe-image → start-seid-once → await-seid-start → mark-not-ready → record(AfterExit)
none ← any (release)	observe-image → mark-ready → record(none)

Start once supports coordinated chain recovery with one commit per validator. The commit carries the recovery image, a ConfigMap with halt-height, and the change from Immediate to AfterExit. The pod rolls, seid starts once, and seid parks when it exits at the halt height.

• The gate closes again a few seconds after seid start appears. On atlantic-2 and pacific-1, seid needs minutes to load state, so it cannot commit a block first.
• A small harbor chain loads faster. A rehearsal there checks the flow, not the timing.

Details

• Start guard (internal/task/start_guard.go). While a hold is set, mark-ready fails as terminal on every path, including the MarkReady SeiNodeTask. start-seid-once submits the same sidecar mark-ready, but only a pending reset blocks it.
• While a hold is requested:
  • The reset plan and the init plan park instead of releasing. Their final mark-ready becomes record(Immediate).
  • The image-update plan drops mark-ready.
  • The planner builds no readiness reapproval.
  • The StatefulSet still applies, so a template change rolls the pod and the new pod stays parked.
• Planner order (staticConfigPlanner.buildRunningPlan): reset, then hold change, then template drift, then reapproval.
• New sidecar task. await-seid-start is read-only. It polls /proc every 250 ms for seid start.
• New controller tasks. start-seid-once and record-maintenance-hold.
• Status.
  • MaintenanceInProgress is always present. Its reasons are NotApplicable, NotHeld, HoldPending, Held, and Armed.
  • Each transition records an event: MaintenanceHeld, MaintenanceArmed, or MaintenanceReleased.

Not in this PR (spec 010 "Out of scope")

• Production pods/exec RBAC. The platform draft is not pushed yet, because the RBAC piece needs your decision.
• A sign-state regression check on release.
• A check on release for an exec'd process. The runbook rule covers it (sei-protocol/runbooks#162).

Needs sign-off

The API fields are listed in #590:

• spec.maintenance.hold, an enum of Immediate and AfterExit
• status.maintenanceHold
• the MaintenanceInProgress condition
• the new sidecar wire type await-seid-start. Wire types are a published contract.

Tests (each names its spec requirement)

Spec item	Test
Req 1 / SC-001	internal/controller/node/envtest/maintenance_validation_test.go
Req 2.1–2.3, 2.7, 4 / SC-002	TestHoldPlans (6 transitions)
Req 2.4 / SC-003	TestHold_StaleUpdatePlanCannotReleaseGate, TestStartGuard_Hold, TestReconcile_MarkReady_RefusedWhileHeld
Req 2.3 start once	TestHold_StartOncePassesGuard, TestAwaitSeidStart_*
Req 2.5, 2.6, 2.8	TestHoldInEffect_NoPlanNoReapproval, TestHold_UpdatePlanCarriesNoMarkReady
Req 2.9 / User Story 5	TestHold_InitPlanParks
Req 3 / SC-004	TestHold_ResetParks, TestRelease_WithResetPending
Req 5 / SC-005	TestResolveMaintenance

SC-006 and SC-007 (harbor: the hold survives a pod delete and a controller restart; AfterExit at halt-height) are judgement checks. They are not run yet.

Independent review: fixes in bc05b03

Finding	Severity	Fix
A hold set while an init plan ran made the guard refuse that plan's mark-ready. An init plan failure is terminal, so the node went Failed.	High	The hold half of the guard acts only on a Running node. seid starts once, then the hold plan stops it (TestHold_SetMidInitDoesNotFailNode).
await-seid-start had no time limit. A pod roll between start-seid-once and seid starting held the plan slot forever.	High	The task fails after 2 minutes, and the plan is built again (TestAwaitSeidStart_FailsAfterTimeout).

This branch merges #594's fixes (b24613f). It uses a merge commit, not a rebase, so nothing was force-pushed.

Deploy and rollback order

• Deploy the sidecar image first, then the controller. An older sidecar rejects await-seid-start.
• Before you roll the controller back, release every hold and let every hold plan finish. An older controller has no hold concept. It would build a readiness reapproval and start seid under the hold. It also fails a persisted hold plan with UnknownTaskTypeError.

Verification

All commands ran with env -u GOROOT GOTOOLCHAIN=go1.26.0.

Check	Result
gofmt -l (3 modules)	no output
go vet ./... (3 modules)	ok
golangci-lint v2.12.1 --new-from-merge-base=origin/main	0 issues. in each module
make test	exit 0; 29 packages ok
envtest (4 suites, k8s 1.34)	all ok
make tidy-check, make verify-generated, make build	pass

Refs: PLT-1390, PLT-1389

🤖 Generated with Claude Code

[bdchatham]

@seidroid review

[seidroid]

This adds spec.maintenance.hold, which parks seid at the sidecar start gate through hold/start-once/release plans, a hold-aware start guard and a new await-seid-start sidecar task. Two paths, both first raised by codex and both confirmed against the tree, can leave seid running while MaintenanceInProgress reports Held, which the API documents as "exec work on the data is safe"; no other scout readings were supplied.

Blocking

2 findings on the changed lines, as inline comments.

1 nit, not posted on the code

• internal/planner/planner.go:311 — A held init plan now carries record-maintenance-hold, so classifyPlan labels it maintenance-hold rather than init in the plan-duration metrics.

_{seidroid review · decision request_changes · session 1edca46d6e9a4c1baccab3d6484925cc · turn resp_claude_382e751c1f8157c81d1a8f3c7f2b0258 · item 29e7f56705f85536bdaf8163b364e5ca}

Findings: 2 blocking | 0 non-blocking | 2 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since the last review, start-seid-once has its own guard (startOnceBlocked), which refuses unless the request is still AfterExit. A hold now goes HoldPending while a hold plan runs or the gate is observed open, and buildHoldPlan rebuilds a hold whose gate the probe reports open, so both earlier blockers are fixed (checked against the tree: the probe runs fresh before every plan build, and an unreachable sidecar is Unknown, not open). Codex's one new reading, that open-gate recovery skips the unequal want/have state, holds and is kept as a non-blocking suggestion; no other scout readings were supplied.

Non-blocking

1 finding on the changed lines, as inline comments.

1 nit, not posted on the code

• internal/task/task.go:239 — The registry comment says start-seid-once is "start-guarded by resets only", but startOnceBlocked now also refuses any request other than AfterExit.

_{seidroid review · decision approve · session 1edca46d6e9a4c1baccab3d6484925cc · turn resp_claude_85a8aec48b4701707fd987704c40f2d9 · item 540c8df73b5a535b9175099e6b19daa6}

Findings: 0 blocking | 1 non-blocking | 1 posted inline

Superseded: the latest review found nothing blocking in this change.

[bdchatham]

@seidroid review

[seidroid]

Since the last pass, buildHoldPlan closes a gate the probe reports open whenever a hold is both requested and in effect, even if the two differ: it re-applies the hold in effect first and retries start-once on the next plan, which fixes the open thread (stop-seid treats an already-stopped seid as success, and normal hold-from-none planning is unaffected); the stale registry comment in task.go is fixed too. Nothing blocks; codex found nothing this round and I found nothing new.

_{seidroid review · decision approve · session 1edca46d6e9a4c1baccab3d6484925cc · turn resp_claude_516ce0feefc7b14123fcacea56bd2c8f · item 322cd692d6f85fce9bd33d576d8df6bf}

Findings: 0 blocking | 0 non-blocking | 0 posted inline

[bdchatham]

@seidroid review

[bdchatham]

@seidroid review

[cursor]

PR Summary

High Risk
Changes core node lifecycle: start-guard behavior, task plans, and a new published sidecar task type; mis-ordering sidecar vs controller deploy or rolling back with holds still set can start seid under a hold.

Overview
Adds maintenance hold for ConfigMap-configured SeiNodes: spec.maintenance.hold (Immediate / AfterExit) parks seid at the sidecar start gate while the pod and data volume stay up for kubectl exec work. Admission requires spec.nodeConfig; status.maintenanceHold tracks the hold in effect, with MaintenanceInProgress condition reasons and Kubernetes events on hold/arm/release.

The start guard now blocks mark-ready (and stale plans / MarkReady tasks) while a hold is set on a Running node, with a dedicated start-seid-once path for AfterExit transitions from Immediate. The planner orders reset → hold change → template drift → reapproval, swaps trailing mark-ready for record-maintenance-hold on init/reset when held, strips mark-ready from image rolls, and can append a clear-hold step when release coincides with a pending reset. New sidecar wire type await-seid-start (2m timeout) supports the start-once sequence; startDeferred logic is tightened so a trailing record step after a refused start does not mis-classify reset failure.

Spec 010 acceptance criteria are nudged for release+reset and condition wording; CRD/manifests and deepcopy follow the API changes, with broad planner/controller/sidecar tests.

^{Reviewed by Cursor Bugbot for commit b4b8fa6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[seidroid]

Nothing material has changed since the last pass: the diff, the merge commit (1a76b1c), the PR description and the base standards are all identical, and the three earlier threads remain correctly resolved. Codex repeated both claims; I keep the first as the open blocker below (releasing a hold together with a data reset reports Held through the reset) and again drop the second, because spec 010 User Story 5 says a node created with AfterExit initializes parked and is then started once, which is what the code does.

Blocking

1 finding on the changed lines, as inline comments.

_{seidroid review · decision request_changes · session 1edca46d6e9a4c1baccab3d6484925cc · turn resp_claude_8dbde396c5b2e822e4376ac7007a2a00 · item 643a3cb6f9b1551387fce53792acb69f}

Findings: 1 blocking | 0 non-blocking | 1 posted inline

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5c089bf. Configure here.}

[cursor]

Hold condition lags after plan

Medium Severity

MaintenanceInProgress is written only at the start of reconcile, before the sidecar probe and before the hold plan can finish. After record-maintenance-hold commits, the condition can stay HoldPending—and even claim the start gate is still open—because it still reads the pre-plan SidecarReady value. MaintenanceHeld / MaintenanceArmed then wait for a later poll. Data reset avoids this with observeTerminalDataResetPlan; the hold path has no equivalent.

Additional Locations (2)

• internal/controller/node/controller.go#L250-L251
• internal/planner/planner.go#L259-L268

 

^{Reviewed by Cursor Bugbot for commit 5c089bf. Configure here.}

[bdchatham]

@seidroid review

[seidroid]

Since the last pass, a reset plan that also releases a hold still in effect now ends mark-ready → record-maintenance-hold("") (clearHoldAfterRelease), ResolveMaintenance reports any requested-vs-in-effect mismatch as HoldPending (a pending release included), and startDeferred handles the new trailing step; spec 010 Req 4.2 and 5.2–5.3 are updated to match, so the open blocker is fixed. Codex's one reading repeats its AfterExit-at-creation claim and is dropped again, because spec 010 User Story 5 says such a node initializes parked and is then started once, which is what the code does.

_{seidroid review · decision approve · session 1edca46d6e9a4c1baccab3d6484925cc · turn resp_claude_d3ae227ddd1516dbe15d6b6942084671 · item 845cc174ab135737803cdd5c2bec85fc}

Findings: 0 blocking | 0 non-blocking | 0 posted inline

Superseded: the latest review found nothing blocking in this change.