feat(seinode): declarative data reset with a start guard (spec 009, PLT-1342)

[bdchatham]

Summary

This PR implements spec 009 (#590). A nodeConfig SeiNode can now reset its data with one GitOps commit, and nothing can start seid while a reset is pending.

• API. spec.dataResetGeneration is the request. status.dataResetGeneration is the last value the controller handled. CEL rejects a lowered or removed counter, and a counter on a node without nodeConfig. While the node is not Running, the handled counter follows the spec, so creating a node never wipes it.

• Reset plan. It comes before any other Running-phase plan:

  1. observe-image, which waits for the rollout
  2. mark-not-ready
  3. stop-seid
  4. reset-data, with 5 retries and backoff
  5. record-data-reset
  6. mark-ready

  The same sequence runs whether or not the pod rolled. The controller does not treat a configRef change as drift (planner.go:852-854).

• Start guard (internal/task/start_guard.go). The mark-ready deserializer checks the target SeiNode at submission. The plan executor and the MarkReady SeiNodeTask both reach that deserializer, with the SeiNode as cfg.Resource. While a reset is pending, mark-ready fails as terminal and submits nothing. A plan built before the counter bump fails, and the planner builds the reset plan next.

• Sign state (sidecar/tasks/reset_data.go).

  • The wipe skips priv_validator_state.json, so a crash during the wipe cannot leave a rerun that writes a zero state.
  • The zero state is written only when the file is absent, and it is fsynced.
  • The reset refuses while a seid or seidb process runs in the pod, found through /proc/*/comm.
  • The reset refuses when config.toml sets a non-default [priv-validator] state-file, or cannot be parsed.

• Status.

  • DataResetInProgress is always present. Its reasons are NotApplicable, NoResetRequested, ResetPending, ResetRunning, ResetComplete, and ResetFailed.
  • ResetFailed stays through retries until a reset succeeds.
  • Each transition records an event: DataResetStarted, DataResetComplete, or DataResetFailed.

Behaviour change

reset-data used to replace the sign state with a zero state. Now it keeps an existing file. This also applies to the SeiNodeTaskWorkflow StateSync recipe, which only targets non-validators, where the file is unused.

Needs sign-off

The new API fields are listed in #590. Please approve their shape before this PR merges.

Tests (each names its spec requirement)

Spec item	Test
Req 1 / SC-001	internal/controller/node/envtest/dataresetgeneration_validation_test.go
Req 1.5 / SC-002	TestDataReset_NotRunningAbsorbsCounter
Req 2 / SC-003	TestDataReset_PlanOrder, TestDataReset_PrecedesUpdateAndReapproval
Req 3.1–3.2 / SC-004	TestDataReset_StaleUpdatePlanCannotReleaseGate, TestMarkReady_StartGuard
Req 3.3 / SC-005	TestReconcile_MarkReady_RefusedWhileResetPending
Req 2.4 / SC-006	TestDataReset_NoPlanWhenHandled
Req 4 / SC-007	TestResetData_KeepsSignStateAndKeys, TestResetData_RerunAfterPartialWipeKeepsSignState, TestResetData_StateFilePath, TestResetData_RefusesWhileDataUserRuns
Req 5 / SC-008	TestDataReset_ConditionTransitions, TestDataReset_FailureStaysReadableUntilSuccess
Counter rises during a reset	TestDataReset_CounterRisesDuringReset

SC-009 (harbor state sync) and SC-010 (runbook) are judgement checks. They are not run yet. The runbook is sei-protocol/runbooks#162.

Independent review: fixes in 4c8dc77

Finding	Severity	Fix
The sign-state guard read only [priv-validator] state-file. sei-config also writes [priv_validator] state_file.	High	reset-data checks both spellings, and refuses a value it cannot read (TestResetData_StateFileSpellingsAndShapes).
An older sidecar would run the old reset-data and zero the sign state, if the controller shipped first.	High (found during the fix)	New wire type reset-data-keep-sign-state, which an older sidecar rejects.
A guard refusal of the final mark-ready read as ResetFailed, although the wipe ran.	Medium	It now reads ResetComplete with "start deferred" (TestDataReset_DeferredStartIsNotAFailure).
The sidecar can rehydrate a stranded mark-ready after a crash.	Medium	Documented, not fixed. The window is one no-op handler, and the sign state still guards against a double sign (spec 009 Assumptions).
A status flush conflict could cost one extra wipe.	Low	Checked and not real. Task IDs are deterministic, so the rerun polls the same completed reset-data.

Deploy and rollback order

• Deploy the sidecar image first, then the controller. The reset plan submits reset-data-keep-sign-state, a wire type that only this sidecar accepts. If the controller ships first, an older sidecar rejects that type. The reset then retries with seid held, instead of running the old reset-data, which zeroes the sign state.
• Roll back the controller only after every reset plan finishes. An older controller does not know record-data-reset or reset-data-keep-sign-state, so a persisted reset plan fails with UnknownTaskTypeError.

Verification

All commands ran with env -u GOROOT GOTOOLCHAIN=go1.26.0, because gvm on this machine pins GOROOT to 1.25.6.

Check	Result
gofmt -l (3 modules)	no output
go vet ./... (3 modules)	exit 0
golangci-lint v2.12.1 --new-from-merge-base=origin/main (3 modules)	0 issues. each
make test	exit 0; 29 packages ok
envtest (4 suites, k8s 1.34)	all ok
make tidy-check, make verify-generated, make build	pass
staticcheck ./...	findings only in files this PR does not touch (deprecations, one unused test helper)
govulncheck ./...	28 / 16 / 28 findings in the Go 1.26.0 stdlib and pinned deps. These are the same counts as other branches. go.mod is unchanged.

Refs: PLT-1342, PLT-1389

🤖 Generated with Claude Code

[bdchatham]

@seidroid review

[seidroid]

Adds spec.dataResetGeneration, which lets a nodeConfig SeiNode wipe its data/ directory through a dedicated reset plan. A start guard on every mark-ready path keeps seid held while a reset is pending, and a new sidecar wire type keeps priv_validator_state.json through the wipe. I approve because the guard, the counter handling, the plan ordering and the sign-state keep hold up against the tree and nothing blocks; two suggestions and one nit remain. Of codex's two findings I kept neither. sei-tendermint maps the sign state only from [priv-validator] state-file, so seid never reads a root-level priv_validator_state_file. tomlpatch.ReadTOML returns an empty map when the file is missing, so TestResetData_MissingDataDirIsSuccess still passes.

Non-blocking

1 finding on the changed lines, as inline comments.

• While a reset is pending, a MarkReady SeiNodeTask now fails terminally, which changes how that kind behaves. CLAUDE.md requires updating the seinode-task design doc in bdchatham-designs for such a change, and the gov-ops skill cites its headings. Nothing in this PR shows that the doc's MarkReady section records the new refusal.

1 nit, not posted on the code

• internal/controller/node/controller.go:336 — This compares only the reconcile-start condition with the end state. On a deferred start, handleTerminalPlan sets ResetComplete and markDataResetStarted overwrites it with ResetRunning in the same reconcile. DataResetComplete is therefore never recorded for the reset that did run, although the PR says every transition records an event.

_{seidroid review · decision approve · session e5b46a97d73e48efa9f57026ab6ebb6b · turn resp_claude_82e1b3eb203b5e9703a9638e59258940 · item 8fc27e85400e56319c3d4a4e5dc3d814}

Findings: 0 blocking | 2 non-blocking | 1 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since the last review, a stale update plan that the start guard defers now ends with NodeUpdateInProgress=False/UpdateComplete instead of UpdateFailed. A counter that rises during a reset now reads ResetPending, and the move from one ResetRunning to the next now records DataResetStarted; both are tested, so the open thread and the events nit are addressed. Codex repeated its two findings, and I dropped both again. seid's sei-tendermint config reads the sign-state path only from [priv-validator] state-file, never from a root-level priv_validator_state_file. tomlpatch.ReadTOML returns an empty map when config.toml is missing, so TestResetData_MissingDataDirIsSuccess still passes. That leaves one earlier non-blocker and nothing blocking.

Non-blocking

• While a reset is pending, a MarkReady SeiNodeTask now fails terminally, which changes how that kind behaves. CLAUDE.md requires updating the seinode-task design doc in bdchatham-designs for such a change, and the gov-ops skill cites its headings. Nothing in this PR shows that the doc's MarkReady section records the new refusal.

_{seidroid review · decision approve · session e5b46a97d73e48efa9f57026ab6ebb6b · turn resp_claude_0ae012d2b2c1ad5a520f9ed315d5ad8f · item a602b5c6725955eeaddba2db8b32c098}

Findings: 0 blocking | 1 non-blocking | 0 posted inline

[bdchatham]

On the design-doc note: sei-protocol/bdchatham-designs#140 records the new MarkReady refusal in the kinds table and as gotcha 4. It covers both a pending reset (this PR) and a maintenance hold (#595). No heading changes.

[cursor]

PR Summary

High Risk
Touches chain data deletion, validator double-sign protection, and the seid start gate—incorrect ordering or sign-state handling could cause data loss or unsafe validator restarts.

Overview
Adds a GitOps-driven data reset for nodeConfig SeiNodes: bump spec.dataResetGeneration (monotonic, CEL-gated) to request a wipe of <home>/data/ while seid stays at the sidecar start gate; completion is status.dataResetGeneration >= N, not a condition alone.

The controller runs a reset plan before any other Running-phase work (observe-image → hold/stop → reset-data-keep-sign-state → record counter → mark-ready), maintains DataResetInProgress status and Kubernetes events, and wraps mark-ready with a start guard so stale update plans and MarkReady tasks cannot start seid on uncleared data.

Sidecar behavior changes: reset-data now preserves priv_validator_state.json (new wire type for version safety), refuses unsafe conditions (live RPC, seid/seidb in the pod, non-default sign-state path in config.toml), and only writes a zero sign state when the file is absent.

^{Reviewed by Cursor Bugbot for commit 865c4e8. Bugbot is set up for automated code reviews on this repo. Configure here.}