Skip to content

feat(env): set up the infra envs from their published images, building only what isn't published - #277

Merged
earakely-scale merged 5 commits into
mainfrom
feat/pull-published-infra-images
Oct 21, 2026
Merged

earakely-scale merged 5 commits into
mainfrom
feat/pull-published-infra-images

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Installs now pull the infra images the release workflow publishes on GHCR instead of building them on this host. Remote VMs pull them too, rather than taking a push of the host's tarballs, or rebuilding all four when the host is arm64.

Bootstrap (env/bootstrap.py). These rules apply on local stores only.

The store's infra env Every one of the kind's images published for these sources Run
missing, or built from other inputs yes / no registers the published images / builds on this host, as before
stock, built here from these sources yes / no switches: a new env version naming the published images / keeps it
already the published images for these sources — nothing
built from someone's own Dockerfile, or stores aren't local — never touched
  • Digest: the one the release wheel records (published_images.json). An install with no record (editable, git, CodeArtifact) uses the one GHCR serves under the source tag; it's pinned when the image is registered.
  • All or nothing: a kind's images are all pulled or all built, so a service-db with one image unpublished is built.
  • Docker: registering needs no Docker on this machine, so only the envs a run builds need it.
  • Output: the run prints one line per env it registers or switches. The dry run says would build, would pull its published image(s), or would switch to ….

Images (DockerImageArtifact)

  • New optional field image_digest: the registry's digest of the built image, beside source_digest, the digest of its sources. It's written as null otherwise, as platform and source_digest were when they were added.
  • A published image is named by its source tag, ghcr.io/scaleapi/agentenv-<name>:src-<hex>. Compose and docker run start containers from that tag, so nothing that starts containers changed.
  • put_ref, given the build context and the digest the tag is published under, registers the tag as image_name with the digest beside it, and uploads the context. Without them, it registers a ref as before. pinned_name is tag@image_digest.

Loading (VmSandbox.load_docker_images)

  • Pull: an image with a digest is pulled by tag@digest, then tagged with its tag.
  • Rate limits: a 429 is retried after 10 s and then 30 s. That's shorter than the release builds' waits because a VM that can't pull falls back to a build of about a minute. docker pull doesn't show Retry-After, so the waits are fixed.
  • Fallback: a VM that still can't pull builds the image from its stored build context under the same tag. If that fails too, the error names both failures.
  • Local provider: it skips the pull when its Docker already holds the tag, so a run after the first needs no registry.
  • Modal: Modal containers pull by the digest.
  • Other providers: every VM provider inherits all of this from VmSandbox, since they override only how a tar.gz reaches the VM. A gateway stack runs in containers only on Modal, the one provider with a private network. One path isn't pinned: an infra env deployed on its own as a server (env_server_provider) on a provider other than Modal is pulled by its tag, without the digest and with no fallback build. Published tags are never moved, and infra envs aren't deployed that way in practice. Pinning it would mean checking each provider's by-name start (some turn a name into a template), so it's left for now.

No defensive code was added for an image with a digest but no build context; nothing writes one.

Tests

  • Unit tests:
    • every row of the bootstrap table, with no Docker needed to register;
    • the lookup, release record first, remembering only digests the registry actually served;
    • pull then tag, the fallback build, rate-limit retries, and the error when both pull and build fail;
    • the local skip-if-present, Modal pulling by digest, and the dry-run wording.
  • The unit conftest stubs the registry lookup, so unit tests see nothing as published and never reach a registry.
  • Full unit tier: 8,852 passed. test_an_armed_time_trigger_keeps_firing_after_a_client_leaves fails intermittently under -n auto and passes alone; it's timing-based and untouched here.

Acceptance on admin VMs (amd64 and Graviton: the timing harness, a burst, and the fallbacks) is in the comment below. A modal_vm env deploy took 46–48 s from Linux (baseline 64–76 s) and 46–62 s from arm64 (a Mac's took 108–131 s). 20 concurrent deploys all passed with no 429s.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issue was found.

Summary

Infra environments use published images when every image for their installed sources is available. Otherwise, they build locally.

  • Local runs use published images for stock environments when a complete set is available.
  • Published image records keep a registry digest beside the tag used to start containers.
  • Sandboxes pull published images by digest and build them when a pull fails.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Check required infra environment] --> B{Needs setup?}
  B -->|No| C[Keep current version]
  B -->|Yes| D{All images published?}
  D -->|No| E[Build locally]
  D -->|Yes| F[Register tags, digests, and build contexts]
  F --> G[VM pulls by digest]
  G -->|Pull succeeds| H[Tag image for containers]
  G -->|Pull fails| I[Build from stored context]
Loading

Reviews (5) · Last reviewed commit: "refactor(env): name the infra setup for ..." · Reviewed by Greptile

…g only what isn't published

A run on local stores now registers a kind's infra images from the ones the release workflow published for the sources
installed, when GHCR has every one of them, instead of building them on this host. The digest is the one the release
wheel records or, with no record, the one GHCR serves under the content tag, pinned when the image is registered. A
stock env built here from unchanged sources switches to the published images once they exist; an env built from
someone's own Dockerfile, or in stores that aren't local, is never touched. Registering needs no Docker here.

DockerImageArtifact gains an optional digest: the image is named by its content tag, which its containers start from,
and pulled by the digest, then tagged. A VM that can't pull it, after waiting out a rate limit twice, builds it from the
build context the artifact keeps, under the same tag, so compose and docker run find it either way. The local provider
skips the pull when its Docker already holds the tag, so later runs need no registry; Modal containers pull by the
digest. The dry run says which envs it would build, pull, or switch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@earakely-scale

earakely-scale commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Acceptance on admin VMs. Each run is the timing bundle's env task: the items MCP server through the gateway, with local Postgres. The baselines are the 2026-10-20 timing pass on v0.9.1336.

Linux amd64 (m5.2xlarge) Graviton arm64 (m7g.2xlarge), standing in for a Mac
modal_vm deploy, fresh stores (run total) 47.7 s, 47.6 s (baseline 64–76 s) 62.4 s (an 8 s VM create), 46.1 s (Mac baseline 108–131 s)
registering the gateway and service-db 0.1 s, no host build 0.6 s
the VM getting the 4 infra images 3.4–4.5 s pull (baseline: 24.6 s tarball push) 3.0–5.3 s pull (baseline: 67–89 s rebuild in the VM)
local provider: first run, then a second with ghcr.io blocked 44.0 s, then 24.1 s with no pull 48.8 s, then 23.8 s with no pull
gateway registered under a digest GHCR doesn't serve pull failed, built from its context in 20.2 s, deploy passed (68 s) built in 25.4 s, deploy passed (83 s)
infra built on the host, then switched dry run would switch; the run switched (58.9 s); nothing left to set up after the same (44.8 s)
20 concurrent modal_vm deploys 20/20 passed in 43–61 s; no 429s, no fallback builds not run

Notes

  • Both hosts ran Docker 29.1.3 on the containerd image store. CI's integration-local-slow covered Docker 28 on the classic store: it registered the published gateway and service-db images, with no host build and no fallback, and its deploys ran on them.
  • ghcr.io blocked during setup: the dry run says would build, but the host build itself fails, because service-db's db-web base (ghcr.io/sosedoff/pgweb) and db-mcp builder (ghcr.io/astral-sh/uv) also come from ghcr.io. So a GHCR outage already blocks building service-db, independent of this PR. I tested the switch by hiding publishing from the first setup in-process instead.
  • No setup needed to pull, checked on fresh VMs: five new VMs had their distro's stock Docker and no registry login or config. They were Amazon Linux 2023 on x86 and arm64 (Docker 25.0), Ubuntu 24.04 on x86 in us-west-2 and in us-east-1 (Docker 29.1, containerd store), and Debian 12 on arm64 (Docker 20.10). Each made the anonymous lookup of all five published tags and got the recorded digests, then pulled each image by tag@digest for its own platform, from its own egress IP. The VMs are deleted.
  • Modal containers, Sail and smol_vm: the same env task ran twice on each, from an m8i.2xlarge with nested virtualization, with every run passing. On Modal's container path, Modal pulled the gateway by tag@digest (13.9–59.8 s per run). Local Postgres runs the stock Postgres image in that mode, so the service-db images aren't used there. On sail_vm, the Sailbox pulled the 4 infra images by digest in 27–28 s, with deploys of 75–79 s. On smol_vm, the guest pulled them in 20–23 s, with deploys of 58–61 s. No run fell back to a build or hit a rate limit. No Modal sandbox was left running, and the Sail key and Modal token never appear in the results.
  • Cleanup: no Modal sandboxes are left in the test app, and no token text appears in any result. The VMs, security group and key pair are deleted.

@earakely-scale
earakely-scale marked this pull request as ready for review October 21, 2026 16:03
@earakely-scale
earakely-scale requested a review from a team as a code owner October 21, 2026 16:03
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

@greptileai review

Comment thread src/agent_env/env/bootstrap.py Outdated
Comment thread src/agent_env/providers/sandbox_providers/modal_sandbox.py
earakely-scale and others added 3 commits October 21, 2026 09:20
…d drop a stale Modal pin

A switch, a pull or a rebuild of an infra env in the store wrote its new version with only the metadata the put
records afresh, dropping what was given with --metadata. The new version now keeps the env's own keys and records
the generated ones (who built it, with which agent-env, from which Dockerfile, the git state, the build inputs) anew.

Modal kept the digest it was told for an image name after the same name was prepared again without one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ge_digest

The artifact already has source_digest, the digest of the sources an image is built from; a bare digest beside it
didn't say which one it was. image_digest is the registry's digest of the built image, as Docker and the registries
call it. The field hasn't been released, so no stored document has it under the old name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…build context and digest

A published image is an image already in a registry, which put_ref registers, so it takes the two things that make
one different instead of a constructor of its own: the build context, uploaded for a sandbox that can't pull the image
to build it from, and the digest the tag is published under. Given them, the tag is recorded as image_name and the
digest beside it, as given; without them, a ref is registered as before. The published path now also gets put_ref's
checks that a ref names its registry and isn't on this machine's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

@greptileai review

…t of facts in one place

- InfraBuild and infra_to_build are InfraSetup and infra_to_set_up: they pull and switch too.
- How a run sets an env up is a Setup enum carrying what the run and a dry run say, in place of three strings and a
  reason constant; the digests it pulls and the metadata it keeps are plain mappings.
- ServiceDBEnv.images lists its three images, which bootstrap and the local Postgres store read instead of each
  listing them; the keys detected() names are one tuple, and a test fails if detect_env_metadata records one it
  doesn't name.
- The docstrings of put_ref, infra_to_set_up, ensure_default_envs and load_docker_images are down to their point.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 3d83f45 into main Oct 21, 2026
14 checks passed
@earakely-scale
earakely-scale deleted the feat/pull-published-infra-images branch October 21, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant