Skip to content

feat(seinode): wake drift-deferred nodes when a roll slot frees (spec 012 Req 1.5) - #606

Merged
bdchatham merged 2 commits into
mainfrom
brandon2/plt-1399-wake-deferred
Oct 8, 2026
Merged

bdchatham merged 2 commits into
mainfrom
brandon2/plt-1399-wake-deferred

Conversation

@bdchatham

@bdchatham bdchatham commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Closes the last PLT-1399 acceptance criterion: a deferred node starts within one requeue interval after a slot frees.

A drifted node that waits for a roll slot re-checked only on its 30-second status poll. The node controller also reconciles one node at a time, so the wait was the poll plus the queue. In the 2026-10-08 rollout of cff50d3, the measured wait from a freed slot to the next start was:

Cell Namespace (nodes) Median Longest
prod-apne1 giga-testnet-0 (12) 1s 27s
prod-use2 arctic-1 (12) 10s 45s
prod-euw1 giga-testnet-0 (12) 26s 29s
prod atlantic-2 (7) 23s 52s
prod giga-testnet-0 (12) 28s 51s

Change

  • New watch (spec 012 Req 1.5). slotReleased passes a SeiNode event that can free or add a slot: the node leaves NodeUpdateInProgress=True, its spec.paused flips, its phase changes (the slot count counts Running, unpaused nodes), or it is deleted. deferredPeers maps that event to every node in the namespace whose NodeUpdateInProgress reason is UpdateDeferred. Each woken node runs the slot decision again with the existing uncached read, so a stale cache costs at most one extra reconcile.
  • planner.ReasonUpdateDeferred is now exported, so the node controller can match it.
  • Spec 012 gains Req 1.5, SC-003, and a known limit: a woken node still waits behind the nodes queued before it.

The change does not alter the slot decision, the order, or the pin. It only changes when a waiting node re-checks.

Verification

  • TestSlotReleased: completion, failure, pause, a phase change, and deletion wake; start, steady state, and creation do not.
  • TestDeferredPeers: only UpdateDeferred nodes in the same namespace are woken.
  • go test ./internal/... ./cmd/... passes (with envtest). golangci-lint --new-from-merge-base=origin/main reports 0 issues. go vet passes.

After the merge, the rollout is a controller-only bump: the cell sidecar does not change, so no node pod rolls.

Refs PLT-1399.

🤖 Generated with Claude Code

… 012 Req 1.5)

A drifted node that waits for a roll slot re-checked only on its 30-second
status poll. A watch on SeiNode now enqueues every UpdateDeferred node in the
namespace when a node leaves NodeUpdateInProgress=True or is deleted, so the
next node in slot order starts without waiting out the poll.

Refs PLT-1399.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches node controller watches and reconcile fan-out during paced drift rollouts; behavior is additive and does not change budget gating, but incorrect predicates could cause extra reconciles or delayed starts.

Overview
Drift-roll budget (spec 012 Req 1.5): deferred nodes no longer wait only for the ~30s status poll when a roll slot frees. The node controller adds a SeiNode watch with slotReleased (update complete/fail, pause flip, phase change, or delete) and deferredPeers, which enqueues every same-namespace node whose NodeUpdateInProgress reason is UpdateDeferred.

Slot math, order, and pinning are unchanged; only when waiting nodes re-run the slot check changes. planner.ReasonUpdateDeferred is exported so the controller can match deferred peers. Spec 012 documents Req 1.5, SC-003, tightened acceptance wording, and rollout wait stats as a known limit.

Unit tests cover the predicate and peer mapping in driftwake_test.go; planner tests use the exported constant.

Reviewed by Cursor Bugbot for commit e8f19bb. Bugbot is set up for automated code reviews on this repo. Configure here.

@bdchatham

Copy link
Copy Markdown
Collaborator Author

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing material has changed since the last review: the diff matches it byte for byte (275 lines), the merge tree is still at 51215cf, and the PR description is the same, so the change still wakes UpdateDeferred peers when a slot holder leaves NodeUpdateInProgress=True or is deleted, nothing blocks, and the earlier non-blocker on pause and phase changes still stands. The one other reading (codex) again found nothing, which agrees with this review apart from that non-blocker, so it added nothing to keep or drop.

Non-blocking

  • DriftSlot also frees a slot when a holder is paused or leaves PhaseRunning, because either drops it from updating, and the slot count changes as nodes enter or leave Running. slotReleased does not fire for any of these, so a deferred node still waits for its 30s poll in those cases. Either add this to spec 012's Known limits or widen the predicate.

seidroid review · decision approve · session 424d7184ba744a2cbd759b8616d4e1c3 · turn resp_claude_defe8d79c35197cf25835a3753f47bc7 · item 20f0323b6ddb5e79a0ee94116c4070f7

Findings: 0 blocking | 1 non-blocking | 0 posted inline

… changes phase

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bdchatham

Copy link
Copy Markdown
Collaborator Author

Addressed the non-blocking note in e8f19bb: slotReleased now also fires when spec.paused flips or the phase changes, which covers a paused holder, a holder that leaves Running, and a node that enters Running and adds a slot. Req 1.5 in spec 012 says the same. @seidroid review

@bdchatham

Copy link
Copy Markdown
Collaborator Author

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This revision widens slotReleased to also fire when spec.paused flips or status.phase changes, and adds test cases and spec text for both, so every event that can free or add a slot in DriftSlot now wakes the UpdateDeferred peers; that addresses my earlier non-blocker. Nothing blocks on the merge tree (b2e53e1), though the tests were not run because no Go toolchain was available, and the other reading (codex) again found nothing, which agrees with this review, so it contributed nothing to keep or drop.

seidroid review · decision approve · session 424d7184ba744a2cbd759b8616d4e1c3 · turn resp_claude_348c75c5a9114cba4f61d96c2356ab23 · item 08886ed4fe4e5c9c955f21664a3d2703

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@bdchatham
bdchatham merged commit 15d6980 into main Oct 8, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant