Repository navigation
feat: Forgejo backend with one-job runners and idle timeout - #38
Merged
Merged
Conversation
- [[forges]] type = "forgejo" (url, token_env/token_path/token, scopes), polling mode - ForgejoClient: polls org/repo job lists, keys jobs by handle, registers ephemeral runners, maps waiting/running to queued/in_progress, treats null lists as empty - Forge::create_job_runner returns RunnerLaunch (GitHub JIT or Forgejo runner); Forge::runner_kind picks agents that advertise the runner program - proto: CreateRunnerCommand.forgejo (ForgejoRunner), AgentStatus.runner_kinds - proxmox agent installs forgejo-runner (sha256-checked) and runs one-job --handle with label:host labels; tart agent rejects Forgejo commands - rename GitHub-specific scan/verify names, forge-neutral log messages - docs: forge-support status, config template, example agent config
A job held back by a concurrency group stays "waiting" in Forgejo's jobs API, so its one-job runner sits idle (and one for a cancelled job waits forever). - [provisioning] idle_runner_timeout_secs (default 600, min 120, 0 disables) - fleet checks job-bound Forgejo runners past the timeout each tick: running -> leave alone; still waiting or gone -> DestroyRunner, job backs off (timeout, doubling, max 1h) without spending a retry; late events aren't failures - proxmox agent: DestroyRunner accepts the VM name, Destroyed event names the runner - docs: forge-support step 4
An agent drops a finished VM from its status at about the time it reports the
runner's outcome. The coordinator reconciles status as soon as it arrives while
the outcome event waits in the fleet manager's queue, so a finished runner
often looked lost first: a synthetic Destroyed ("VM missing during
reconciliation") recorded a RunnerLost failure, spent a job retry, steered the
job away from the agent, and the real Completed event was then ignored.
reconcile_agent_vms now reports a runner only after its VM has been missing for
MISSING_VM_GRACE (20s).
jfro
changed the title
Forgejo picks a token's scope level by HTTP method, so a read-only token passed the startup check (listing jobs) and only failed later when registering runners. validate() now also deletes runner id 0, which can't exist: 404 when the token may manage runners, 403 when it may not. docs: step 5 results against Forgejo 16.0.5 and 17.0.0
Upstream only releases Linux binaries. A forgejo-runner workflow builds the unmodified upstream tag for linux amd64/arm64, darwin arm64/amd64 and windows amd64 with checksums, the upstream source and license (GPLv3), as a not-latest forgejo-runner-v<version> release. Agents now install a pinned forgejo-runner (13.2.0) from those releases; proxmox gets vm.forgejo_runner_url (URL template) for mirrors, and "latest" means the pinned version. The install script falls back to shasum on macOS.
tart: install forgejo-runner (darwin/linux build by guest OS, shasum on macOS), prepare the job, and run it through a wrapper with the GitHub runner's protocol, launched in Terminal.app on macOS like GitHub jobs. Advertise "forgejo". Waiting for a runner now stops as soon as its VM is destroyed instead of polling the gone VM until the 6-hour limit. proxmox Windows: PowerShell install (Invoke-WebRequest + Get-FileHash against the release .sha256, removed again if --version fails), token file + run-job.ps1, and run, all via -EncodedCommand so cmd.exe and PowerShell SSH shells both work. Errors are caught and printed as one line instead of CLIXML. docs: macOS/Windows runners, what was tested here, and a plan for testing on real hosts
Brings in release-plz, the packaging move, the russh bump and the agent maintenance toggle (#40). Conflicts resolved by keeping both sides: - forge.rs: RunnerKind/RunnerLaunch/ForgejoLaunch alongside IdleRemoval - fleet.rs: idle-runner teardown state alongside the drain state; imports - agent_registry.rs: select_agent_for_kind test alongside the unavailable-agent tests (rebuilt whole from each side) ForgejoClient implements the new Forge::remove_idle_runner: one-job runners are never idle pool runners, so a registered one is reported Busy and kept.
scopes = ["@jfro", "lodge"] covers a user's own repos plus an org's with one token (write:user + write:organization + read:repository); scopes = ["*"] covers the whole instance with a site admin's token (write:admin + read:repository). A user scope only covers repos the user owns, not its orgs'. - RunnerScope gains User and Instance (stored as {"type":"user"} / {"type":"instance"}); GitHub forges refuse them - Forgejo uses /user and /admin endpoints for them; job status is asked at the owning org/user scope, else the instance, else the repo - overlapping scopes list a job once - startup checks: a @user scope must be the token's own user, and org/user/ instance scopes need read:repository (repo names), probed via repository 0 - docs: scopes and tokens table
It's only read by fixed-capacity pools; on-demand runners register where their job is. A polling-only config (e.g. just a Forgejo forge) no longer needs a placeholder, and fixed-capacity mode without one now fails at startup with a clear error instead of being impossible to express.
An idle teardown (DestroyRunner by name) frees the VMID while the runner's lifecycle task is still waiting on the job. Its cleanup then destroyed by bare VMID, which could stop and delete another runner's VM cloned into the reused VMID. destroy_vm now takes the VM's name and only goes ahead if this agent still tracks the VMID under that name and no other destroy has claimed it, and checks Proxmox's name for the VMID before stopping and deleting it.
… torn down A runner command (forgejo-runner one-job, or the GitHub runner) blocked on an SSH session with no keepalive, so after an idle teardown destroyed the VM it could wait forever and never report Completed/Failed. Sessions now send a keepalive every 30s and drop after 3 missed, and the runner command is raced against the VM leaving this agent's tracking (or starting to be destroyed), like the tart agent's vm_gone.
The proxmox and tart agents debug-logged every SSH command in full, and the forgejo-runner job-file write, config.sh and the GitHub JIT config write carry the runner/registration token. Those now go through execute_secret / ssh_exec_secret, which log what the command is and its size, not its text.
${{ inputs.version }} was expanded straight into the bash script, so a
crafted workflow_dispatch input could run commands with contents: write.
It now comes in through INPUT_VERSION. actions/setup-go is pinned to the
v6.5.0 commit instead of the movable v6 tag.
list_runners stopped at the first page shorter than the 50 it asked for, but Forgejo caps pages at MAX_RESPONSE_ITEMS, which an admin can set below 50. Then only the first page was ever read and runners past it were never found or removed. Stop on an empty page instead.
max_vms is INT4 on Postgres, and reading it as i64 panicked in get_agent and list_agents (the agents page, test_agent_version_reported_and_cleared). Read it as i32, which also fits SQLite's INTEGER, with try_get: a bad value is logged with the agent id and the row skipped instead of panicking.
jfro
marked this pull request as ready for review
October 20, 2026 20:59
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f8acd2a07
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+514
to
+518
| Err(e) => { | ||
| warn!( | ||
| "Couldn't look up repository {} for Forgejo job {}: {:#}", | ||
| job.repo_id, job.handle, e | ||
| ); |
There was a problem hiding this comment.
When /repositories/{id} fails transiently for an organization, user, or instance scope, this branch still queues the job with repository = None. check_idle_runners explicitly skips runners without a repository, so a Forgejo job held in waiting can occupy an agent indefinitely instead of being reclaimed after idle_runner_timeout_secs. Skip this job for the current scan, or fail the scope, so it is retried once its required repository metadata can be resolved.
Useful? React with 👍 / 👎.
This was referenced Oct 20, 2026
Member
Author
|
Follow-ups from the review, filed for after merge:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Steps 3–5 of the forge work (docs/forge-support.md): kuiper-forge can poll a Forgejo 15+ instance and run its jobs on single-use runners on proxmox Linux VMs. Draft for early review.
What's in it
[[forges]]entries withtype = "forgejo",url, a token (token_env/token_path/token) andscopes(orgs orowner/repo). Polling mode only: startup rejects a Forgejo default forge in fixed-capacity mode and a Forgejo webhook forge.ForgejoClient(kuiper-forge/src/forgejo.rs): lists…/actions/runners/jobsper scope and keeps thewaitingones, keyed byhandle, with cached repo names. Registers ephemeral runners, removes runners by name (paged), and mapswaiting/runningtoqueued/in_progressfor the fleet manager. Treats Forgejo'snullempty-list response as empty.create_job_runnerreplacesgenerate_jitconfigand returns aRunnerLaunch(GitHub JIT config or Forgejo runner).runner_kindsays which runner program a forge needs.CreateRunnerCommand.forgejo(ForgejoRunner: url, uuid, token, handle; boxed) andAgentStatus.runner_kinds. Forgejo jobs only go to agents that listforgejo, so older agents never get one.vm.forgejo_runner_version, default latest, at least 13.1.0), writes the token to a mode-600 file, and runsone-job --handle … --waitwithlabel:hostlabels. Windows VMs fail clearly. The tart agent rejects Forgejo commands.scan_github→scan_forges,recover_queued_jobs_from_github→recover_queued_jobs,verify_pending_jobs_with_github→verify_pending_jobs, and forge-neutral log messages.Idle runner timeout (step 4)
Forgejo keeps a job held back by a concurrency group as
waitingin the jobs API, so its one-job runner sits idle (and one for a cancelled job waits forever).[provisioning] idle_runner_timeout_secs(default 600, minimum 120,0disables).DestroyRunner.DestroyRunneraccepts the VM name, and theDestroyedevent names the runner.Fix: grace period for missing VMs
An agent drops a finished VM from its status at about the time it reports the outcome. The coordinator reconciles status immediately while the outcome event waits in the fleet queue, so finished runners often looked lost. The result was a synthetic "VM missing" Destroyed event: a RunnerLost failure, a used-up retry, the agent steered away from, and the real Completed event ignored. Affects GitHub runners too.
reconcile_agent_vmsnow waitsMISSING_VM_GRACE(20s) before reporting a runner as missing.Startup token check
Forgejo picks a token's scope level by HTTP method, so a read-only token could list jobs and pass startup, then fail on every runner registration. Startup now also deletes runner id 0, which can't exist. The answer is 404 when the token may manage runners and 403 when it may not; in that case startup stops with the "must own the scope" hint. Same behaviour on 16.0.5 and 17.0.0.
forgejo-runner builds for every platform
Upstream only releases Linux binaries, so
.github/workflows/forgejo-runner.ymlbuilds the unmodified upstream tag for linux amd64/arm64, darwin arm64/amd64 and windows amd64, with checksums, plus the upstream source and license (GPLv3). It publishes them as a not-latestforgejo-runner-v<version>release. Agents install a pinned version (13.2.0) from there. Proxmox getsvm.forgejo_runner_urlfor mirrors, andlatestmeans the pinned version. The install script usesshasum -a 256on macOS. First release: forgejo-runner-v13.2.0.macOS (tart) and Windows (proxmox) Forgejo runners: ready for testing on real hosts
shasumon macOS), prepares the job, and runs it through a wrapper with the GitHub runner's protocol. On macOS it's launched in Terminal.app, like GitHub jobs. The agent advertisesforgejo. Waiting for a runner now stops as soon as its VM is destroyed, instead of polling the gone VM for up to 6 hours (this affects GitHub runners too)..sha256), a token file andrun-job.ps1, and run. Everything goes via-EncodedCommand, so cmd.exe and PowerShell SSH shells both work. Errors print as one line instead of CLIXML.tartand SSH-reachable "VMs": it installed from the published release, ran jobs, and handled idle teardown and retry.docs/forge-support.mdhas a plan for testing on real hosts.Merged with main (release-plz, packaging move, agent maintenance toggle #40)
ForgejoClientimplements the newForge::remove_idle_runner. One-job runners are never idle pool runners, so a registered one is reportedBusyand kept.DestroyRunner, but main's proxmox agent only accepts a numeric VMID. So a drained proxmox VM isn't destroyed, and stays up until stale-VM cleanup (2h by default). This branch's proxmox agent accepts the name. Tart was unaffected.Forgejo user and instance scopes
scopes = ["@jfro", "lodge"]: a user's own repos plus an org's, with one token (write:user+write:organization+read:repository).scopes = ["*"]: the whole instance, with a site admin's token (write:admin+read:repository). A user scope only covers repos the user owns, not its orgs'.RunnerScopegainsUserandInstance. GitHub forges refuse them.@userscope must be the token's own user, and org/user/instance scopes needread:repository.[runner_scope]only in fixed-capacity modeOnly fixed-capacity pools use it, so it's optional now, and fixed-capacity mode without one fails at startup.
Docs
required_labelsapplies to every forge: GitHub addsself-hostedautomatically, Forgejo doesn't, so workflows on both need the required labels.Testing
required_labels = ["self-hosted"], and a 120s idle timeout. Checks:needs:chain succeedsneeds:workflow each got exactly one runner per job, all jobs succeeded, and nothing was left over.Open points
required_labelsis global and defaults to["self-hosted"]; Forgejo setups need to set it (e.g.[]). Should it be per forge?