Repository navigation
feat: mark agents unavailable from the dashboard for maintenance - #40
Merged
Merged
Conversation
An admin can mark an agent unavailable (and available again) from the dashboard or its page. It then gets no new runners: webhook/polling jobs and fixed-capacity pools skip it, while runners already on it finish. The flag is stored on the agent record, so it holds across agent and coordinator restarts, e.g. while updating VM images. Pending jobs whose only matching agents are unavailable say so, and the API reports unavailable/unavailable_since per agent and a count in /status. Closes #39
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8878a38c49
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Marking an agent unavailable only stopped new runners; its existing fixed-capacity runners stayed registered and kept taking jobs, so the host never drained. Each reconcile now retires the idle pool runners of connected unavailable agents: the runner is unregistered from the forge first, which GitHub refuses while it runs a job (busy flag, or 422 on the delete), and only then is DestroyRunner sent for its VM. Busy runners finish and aren't replaced. Runners not registered yet get a 10 minute grace period to finish booting, then they're destroyed too. Drained runners aren't recorded as agent failures. GitHub runner lookups now filter by name instead of reading only the first page of runners, which also fixes remove_runner for scopes with more than 30 runners.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39
Adds a Mark Unavailable / Mark Available toggle for agents, on the dashboard's agent list and in the agent page header, for maintenance like updating VM images.
What an unavailable agent does
DestroyRunnersent for its VM. GitHub refuses to unregister a runner that's running a job (thebusyflag, or a 422 on the delete), so a job can't land on a runner while it's being destroyed.registered_agents.unavailable_atcolumn. The registry loads it at startup and applies it insideregister(), under the agents lock, so an agent restarting mid-maintenance never gets work in between.UI / API
Status column and agent page show an Unavailable badge. The agent page also says since when.
A pending job whose only matching agents are unavailable shows Agents unavailable, not "Agents busy".
/api/v1/agentsreturnsunavailableandunavailable_sincefor each agent./api/v1/statusreturnsunavailable_agents.Toggling is logged with the agent, the admin user, and its active runner count. The "No agent available" warning now includes
unavailable=for each agent.GitHub runner lookups now filter by
name. They used to read only the first page of runners (30), which also brokeremove_runnerfor scopes with more runners than that.Not in this PR
kuiper-forge agentCLI subcommand for this yet. It's dashboard and DB only.Testing
cargo test -p kuiper-forgepasses. Clippy is clean on sqlite and postgres.workflow_jobwebhook. The job showed "Agents unavailable", and the log listedunavailable=truefor each agent.DestroyRunnerwent to A1 and the record was removed.