Skip to content

feat: mark agents unavailable from the dashboard for maintenance - #40

Merged
jfro merged 2 commits into
mainfrom
feat/agent-maintenance-toggle
Oct 20, 2026
Merged

jfro merged 2 commits into
mainfrom
feat/agent-maintenance-toggle

Conversation

@jfro

@jfro jfro commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Closes #39

Adds a Mark Unavailable / Mark Available toggle for agents, on the dashboard's agent list and in the agent page header, for maintenance like updating VM images.

What an unavailable agent does

  • Gets no new runners. Its free capacity counts as 0, so on-demand jobs (webhook and polling) pass it over, and fixed-capacity pools leave it out (both the target and the runner count).
  • Idle fixed-capacity runners on it are drained. On each reconcile the runner is unregistered from the forge first, and only then is DestroyRunner sent for its VM. GitHub refuses to unregister a runner that's running a job (the busy flag, or a 422 on the delete), so a job can't land on a runner while it's being destroyed.
  • Busy runners finish their job, then exit like any ephemeral runner and aren't replaced. Runners that aren't registered yet get 10 minutes to finish booting: one that registers in time is drained as idle, and one that never registers is destroyed. Drained runners aren't recorded as agent failures. Runners made for a specific job (webhook and polling modes) are left alone.
  • Stays unavailable across restarts. The flag is a new registered_agents.unavailable_at column. The registry loads it at startup and applies it inside register(), under the agents lock, so an agent restarting mid-maintenance never gets work in between.
  • Marking it available again wakes the fleet manager, so queued jobs and pool top-ups go out right away instead of on the next tick.

UI / API

  • Status column and agent page show an Unavailable badge. The agent page also says since when.

  • A pending job whose only matching agents are unavailable shows Agents unavailable, not "Agents busy".

  • /api/v1/agents returns unavailable and unavailable_since for each agent. /api/v1/status returns unavailable_agents.

  • Toggling is logged with the agent, the admin user, and its active runner count. The "No agent available" warning now includes unavailable= for each agent.

  • GitHub runner lookups now filter by name. They used to read only the first page of runners (30), which also broke remove_runner for scopes with more runners than that.

Not in this PR

  • No kuiper-forge agent CLI subcommand for this yet. It's dashboard and DB only.

Testing

  • New unit tests: the registry (no selection, capacity, or pools while unavailable, and the flag surviving reconnects) and the auth store (round-trip; status syncs don't clear it).
  • cargo test -p kuiper-forge passes. Clippy is clean on sqlite and postgres.
  • Manual run: dry-run coordinator in webhook mode with two test agents.
    • Marked both unavailable and sent a queued workflow_job webhook. The job showed "Agents unavailable", and the log listed unavailable=true for each agent.
    • Marked one available. The job was dispatched to it right away.
    • Restarted the coordinator. The flag held.
  • Manual drain run: fixed-capacity dry-run with two fake agents whose VMs stay up, one pool runner each. Marked A1 unavailable.
    • Its runner was kept while it was younger than the boot grace (the mock forge reports it unregistered).
    • Once the runner was backdated past the grace, it was drained: DestroyRunner went to A1 and the record was removed.
    • It wasn't replaced, no failure was recorded, and A2's runner was untouched.
  • The forge-side Removed/Busy outcomes are covered by a unit test of the drain decision. The GitHub calls themselves aren't exercised here.

An admin can mark an agent unavailable (and available again) from the
dashboard or its page. It then gets no new runners: webhook/polling jobs
and fixed-capacity pools skip it, while runners already on it finish.
The flag is stored on the agent record, so it holds across agent and
coordinator restarts, e.g. while updating VM images.

Pending jobs whose only matching agents are unavailable say so, and the
API reports unavailable/unavailable_since per agent and a count in
/status.

Closes #39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T18:32:10.706116Z 8878a38 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8878a38c49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread kuiper-forge/src/agent_registry.rs
Marking an agent unavailable only stopped new runners; its existing
fixed-capacity runners stayed registered and kept taking jobs, so the
host never drained. Each reconcile now retires the idle pool runners of
connected unavailable agents: the runner is unregistered from the forge
first, which GitHub refuses while it runs a job (busy flag, or 422 on the
delete), and only then is DestroyRunner sent for its VM. Busy runners
finish and aren't replaced. Runners not registered yet get a 10 minute
grace period to finish booting, then they're destroyed too. Drained
runners aren't recorded as agent failures.

GitHub runner lookups now filter by name instead of reading only the
first page of runners, which also fixes remove_runner for scopes with
more than 30 runners.
@jfro
jfro merged commit 6b2395f into main Oct 20, 2026
3 checks passed
@jfro
jfro deleted the feat/agent-maintenance-toggle branch October 20, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support toggling agents from dashboard as unavailable

1 participant