Repository navigation
Conversation
Prep for `bootc apply-live bound-images`, which needs to know not just which image a bound image definition references, but where the quadlet file that defines it lives (so it can be related to podman's quadlet search directories) and its contents (so definitions can be compared across deployments and materialized elsewhere). Add `query_bound_image_specs` returning that alongside the parsed `BoundImage`, and make `query_bound_images` a thin wrapper over it. Symlink targets are resolved lexically to a root-relative path with the same semantics as the `RESOLVE_IN_ROOT` read of the file (excess `..` stays at the root); no further symlinks are followed. Assisted-by: AI
A customer using logically bound images wants to update them out of band without rebooting, while bootc keeps owning their lifecycle (storage, garbage collection, rollback). This is a scoped first step toward the general `bootc apply-live` in bootc-dev#76. The key observation is that the image bytes are already live: there is a single bootc-owned containers-storage shared by all deployments, and quadlets read it via `additionalimagestore`. What is tied to a deployment is only the definition (the `bound-images.d` symlink and the quadlet it points to), which the running system reads from its own `/usr`. So no overmounting is needed: changed quadlets are written to `/run/containers/systemd/`, which podman gives precedence over `/etc` and `/usr`, systemd is reloaded, and the affected `.container` units restarted. Because `/run` is transient the override disappears on reboot, at which point the staged deployment's content applies and the system converges on its own. A soft reboot preserves `/run`, so the override is cleared when one is prepared. The command refuses to run unless the ostree diff between the booted and staged commits is entirely accounted for by the bound image changes it will apply, so it never silently applies half an upgrade; since the diff doesn't enumerate added/removed directories, those are walked in the deployment roots. The applied state is recorded in `/run/bootc/apply-live/bound-images.json` before any unit is touched, with each unit marked pending until its restart succeeds, so a failed restart (or `--no-restart`) is retried by simply re-running. The state is surfaced in `bootc status`, and its image names become extra GC roots so a `bootc rollback` that discards the staged deployment doesn't prune an image a running container uses. A later staged deployment reverts any overrides from a previous apply that it no longer needs. The verb is a subcommand group (`apply-live bound-images`, alias `lbi`) so that the bare `apply-live` stays reserved for the general feature and further scopes can be added without a new top-level verb. It is hidden and experimental, and ostree-only for now; composefs needs a tree diff API before the scope check can be implemented there. This has unit coverage for the planning, scope check and state logic. The /run write, SELinux labeling, systemctl sequencing, status output, GC root handling, out-of-scope refusal and post-reboot convergence were verified on a CentOS Stream 10 VM by the tmt test in the following commit. Assisted-by: AI
The unit tests cover the planning, scope check and state file logic, but not the parts that only exist on a real host: writing to `/run/containers/systemd` with the right SELinux label, the systemctl sequencing, and the interaction with `bootc status`, garbage collection and reboot. Switch to an image with a bound `.container` that just sleeps, then upgrade to one that only changes the bound image, and walk through `--dry-run`, `--no-restart` (unit still on the old image, state marked pending), a plain re-run (unit restarted onto the new image), and the idempotent no-op. Discarding the staged deployment via rollback and running cleanup checks the live state acts as a GC root. An upgrade with an unrelated file change checks the out-of-scope refusal. After the reboot, the `/run` override and state must be gone and the deployment's own definition in effect. Assisted-by: AI
| const QUADLET_SEARCH_DIRS: &[&str] = &["etc/containers/systemd", "usr/share/containers/systemd"]; | ||
| /// Directories which may be added/removed/changed in an ostree commit diff | ||
| /// while still being in scope for `apply-live bound-images`. Note that | ||
| /// `/etc` is stored as `/usr/etc` in ostree commits. The contents of an |
There was a problem hiding this comment.
Yeah but not in composefs...
There was a problem hiding this comment.
Updated the doc comment to scope that claim to the ostree backend specifically, since the shared storage there is just a plain directory. Noted that composefs lays out image storage differently and isn't covered by this yet. Should I add this as well? Let me know if I get it correctly.
| "architecture" | ||
| ] | ||
| }, | ||
| "LiveBoundImage": { |
There was a problem hiding this comment.
I'm not totally sure about this. A client tool can already discover some of this state by looking in /run directly?
We actually don't have status for non-live bound images either, note...Should we? Maybe. But IMO this also kind of quickly gets into #522 territory.
| } | ||
|
|
||
| /// Read the live bound image state from the host `/run`, if any. | ||
| pub(crate) fn read_state_from_host() -> Result<Option<LiveBoundImages>> { |
There was a problem hiding this comment.
What I'm thinking here is we should try to drive Source Of Truth here closer to the systemd unit files (perhaps with added comments?) instead of inventing a custom JSON file.
There was a problem hiding this comment.
Replaced the JSON state file with state embedded directly in the quadlet files, each one written to /run/containers/systemd/ carries a leading comment recording its checksum, image, and etc, so that directory becomes the source of truth instead of a bespoke state format. There's still one small flat-text list (protected-images.list) for GC protection, but it's not JSON, just a line-per-image list. What do you think of this approach? Any different ideas?
The ELN latest image no longer provides a reliable source for grub-cc. Fetch the architecture-specific RPM from a pinned Fedora Koji build instead. This is temporary until bootupd supports installing grub-cc into the ESP. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Apply two pieces of feedback from cgwalters's review of bootc-dev#2458: - Scope the module doc's "doesn't require mounting anything" claim to the ostree backend, since the composefs backend lays out image storage differently. - Replace the `/run/bootc/apply-live/bound-images.json` state file with state embedded directly in the quadlet files written to `/run/containers/systemd/`: each one now carries a leading marker comment recording its staged checksum, image reference, and whether its unit still needs a restart. That directory is now the source of truth instead of a bespoke JSON file. A small flat-text list of protected image references is still written separately, since a bound image with an unchanged definition has no quadlet override to carry that state, and this list is only used to protect such images from garbage collection. Still under review; not yet addressing the separate comment about whether `liveBoundImages` belongs in `bootc status`/the schema. Assisted-by: AI
There was a problem hiding this comment.
🟡 Changes recommended
Critical state-reconciliation defects can leave stale containers running, expose in-use images to pruning, and bypass administrator configuration, while the integration test fails before reaching its restart and cleanup checks.
9 open issues
Stop previous service when effective unit identity changes · New Retain protected image references until units stop using them · New Reconcile complete Quadlet changes before restart-only path · New Reject local Quadlet conflicts before writing vendor definitions · New Preserve replacement destinations when moving bound Quadlets · New Persist pending stops and reverts across stale override removal · New Read live-bound image state from bootc status JSON · New Skip non-regular entries before reading bootc markers · New Refresh already-live marker to the current staged checksum · New
What changed in this PR
Adds experimental live application of logically bound image definitions on OSTree-backed systems, allowing container updates without rebooting when the staged deployment contains only supported definition changes.
Changes:
- Adds
bootc apply-live bound-images, Quadlet overrides, restart tracking, status reporting, garbage-collection protection, and soft-reboot cleanup. - Documents the interface and limitations, and adds unit and multi-boot integration tests.
- Replaces the GrubCC download stage with conditional downloads of pinned RPMs.
| File | Description |
|---|---|
tmt/tests/tests.fmf |
Registers the live bound-image integration test. |
tmt/tests/booted/test-logically-bound-apply-live.nu |
Tests live updates, deferred restarts, cleanup, and reboot behavior. |
tmt/plans/integration.fmf |
Adds the OSTree-only integration plan. |
docs/src/logically-bound-images.md |
Documents live application and its limitations. |
docs/src/host-v1.schema.json |
Adds live bound-image status definitions. |
Dockerfile |
Downloads pinned GrubCC RPMs only when needed. |
crates/lib/src/status.rs |
Exposes live state in structured and human-readable status. |
crates/lib/src/spec.rs |
Defines live bound-image status types. |
crates/lib/src/lib.rs |
Registers and documents the apply-live module. |
crates/lib/src/deploy.rs |
Protects live-applied images during garbage collection. |
crates/lib/src/cli.rs |
Adds command dispatch and soft-reboot cleanup. |
crates/lib/src/boundimage.rs |
Exposes bound-image paths and contents for comparison. |
crates/lib/src/applylive.rs |
Implements scope validation, overrides, restart handling, and state tracking. |
Review effort: Balanced
| for (path, spec) in staged.iter() { | ||
| match booted.get(path) { | ||
| Some(b) if b.contents == spec.contents => {} | ||
| Some(_) => changes.push(mk(ChangeKind::Updated, spec)?), |
| crate::lsm::atomic_replace_labeled( | ||
| rootfs, | ||
| &dest, | ||
| filemode, | ||
| sepolicy.as_ref(), |
| const quadlet = "/usr/share/containers/systemd/lbi-sleeper.container" | ||
| const run_quadlet = "/run/containers/systemd/lbi-sleeper.container" | ||
| const unit = "lbi-sleeper.service" | ||
| const state_file = "/run/bootc/apply-live/bound-images.json" |
Apply two pieces of feedback from cgwalters's review of bootc-dev#2458: - Scope the module doc's "doesn't require mounting anything" claim to the ostree backend, since the composefs backend lays out image storage differently. - Replace the `/run/bootc/apply-live/bound-images.json` state file with state embedded directly in the quadlet files written to `/run/containers/systemd/`: each one now carries a leading marker comment recording its staged checksum, image reference, and whether its unit still needs a restart. That directory is now the source of truth instead of a bespoke JSON file. A small flat-text list of protected image references is still written separately, since a bound image with an unchanged definition has no quadlet override to carry that state, and this list is only used to protect such images from garbage collection. Also address several correctness issues GitHub Copilot's automated review found in that state-tracking code: - Track the previous unit name across a `ServiceName=` change so the old unit is actually stopped instead of left running alongside the new one. - Merge newly staged protected images into the existing `protected-images.list` instead of replacing it, so an earlier deployment's still-running bound image isn't GC'd out from under it. - Make the fast path compare the full set of expected quadlets, not just checksums, so a retry after a partially-failed apply doesn't mistake incomplete state for complete. - Refuse to overwrite a `/run` quadlet override bootc didn't write itself. - Guard against an Added/Removed pair that target the same `/run` destination (e.g. a quadlet moving between search dirs) from deleting the replacement that was just written. - Always refresh a quadlet's marker checksum, even when its contents are unchanged, so status and the fast path stay accurate across later deployments. - Skip symlinks and other non-regular files when scanning `/run/containers/systemd/`, so a unit masked via `-> /dev/null` doesn't break `bootc status`. - Update the booted integration test to read state from `bootc status --json`'s `liveBoundImages` and `protected-images.list` instead of the now-removed JSON state file. Still under review; not yet addressing the separate comment about whether `liveBoundImages` belongs in `bootc status`/the schema, nor the deeper `/etc`-vs-vendor quadlet precedence question Copilot also raised. Assisted-by: AI Co-authored-by: Cursor <cursoragent@cursor.com>
251c59b to
c9a95fc
Compare
Apply two pieces of feedback from cgwalters's review of bootc-dev#2458: - Scope the module doc's "doesn't require mounting anything" claim to the ostree backend, since the composefs backend lays out image storage differently. - Replace the `/run/bootc/apply-live/bound-images.json` state file with state embedded directly in the quadlet files written to `/run/containers/systemd/`: each one now carries a leading marker comment recording its staged checksum, image reference, and whether its unit still needs a restart. That directory is now the source of truth instead of a bespoke JSON file. A small flat-text list of protected image references is still written separately, since a bound image with an unchanged definition has no quadlet override to carry that state, and this list is only used to protect such images from garbage collection. Also address several correctness issues GitHub Copilot's automated review found in that state-tracking code: - Track the previous unit name across a `ServiceName=` change so the old unit is actually stopped instead of left running alongside the new one. - Merge newly staged protected images into the existing `protected-images.list` instead of replacing it, so an earlier deployment's still-running bound image isn't GC'd out from under it. - Make the fast path compare the full set of expected quadlets, not just checksums, so a retry after a partially-failed apply doesn't mistake incomplete state for complete. - Refuse to overwrite a `/run` quadlet override bootc didn't write itself. - Guard against an Added/Removed pair that target the same `/run` destination (e.g. a quadlet moving between search dirs) from deleting the replacement that was just written. - Always refresh a quadlet's marker checksum, even when its contents are unchanged, so status and the fast path stay accurate across later deployments. - Skip symlinks and other non-regular files when scanning `/run/containers/systemd/`, so a unit masked via `-> /dev/null` doesn't break `bootc status`. - Update the booted integration test to read state from `bootc status --json`'s `liveBoundImages` and `protected-images.list` instead of the now-removed JSON state file. Still under review; not yet addressing the separate comment about whether `liveBoundImages` belongs in `bootc status`/the schema, nor the deeper `/etc`-vs-vendor quadlet precedence question Copilot also raised. Assisted-by: AI Co-authored-by: Cursor <cursoragent@cursor.com>
…ages Verified after merging: passes (342/344; the 2 failures in delta_ostree:: are pre-existing on original/main alone, unrelated to this branch), shows the same pre-existing 2 errors as before the merge, and is clean. Assisted-by: AI
c9a95fc to
85bb17f
Compare
|
Hi @cgwalters ! Joseph handed this off to me so I could dig into bootc a bit more since I've been interested in the project and wanted the chance to get more familiar with the codebase. I made a few changes addressing your comments below and will answer each one. |
Thanks! |
|
@jmarrero-bot review the new commits |
jmarrero-bot
left a comment
There was a problem hiding this comment.
Not ready yet. bd5345b moves the state into the /run quadlet markers and fixes most of what it says it does. But the PR's booted test fails at this head, because the new "already applied" fast path is never taken on the test image. The fast path also drops pending stops, and the walk_files change weakens the scope check. Details are inline.
Prior feedback:
- Addressed: the composefs scoping of the module doc, keeping state in the unit files instead of JSON (cgwalters), and from Copilot: keeping earlier protected images, the search-dir move, skipping non-regular
/runentries, refreshing the already-live marker, and the test readingliveBoundImages. - Partly addressed: the
ServiceName=rename and "reconcile the complete change set". Both work in a single plain run, but are lost after--no-restartor a failed stop (inline). The non-bootc/runoverwrite is refused, but only after other files are already written (inline). - Still open: whether
liveBoundImagesbelongs in status/the schema (cgwalters),/etc-vs-vendor precedence, and Copilot's "persist pending stops/reverts across stale override removal". After--no-restart, the stale marker is deleted and the stop/revert is never retried.
Other:
- 5ba73ae is identical to main's 3c5d235 (
git range-diffshows=), so after the merge it changes nothing. Rebasing instead of merging would drop it, and would also get rid of the merge message, whose backticked commands were eaten by shell substitution. - bd5345b has no Signed-off-by (DCO).
- The merge's conflict resolution in lib.rs and the tmt fmf files looks right, and
cargo xtask update-generated direct --checkis clean.
Tests (16-core RHEL 10 devspace, at 85bb17f):
cargo fmt -- --check -l: clean.cargo test -p bootc-lib --lib: 344 passed, 1 ignored.cargo xtask update-generated direct --check: clean.cargo clippywith the Makefile's flags, using the devspace's clippy 1.92: fails onstr::len(applylive.rs:491, banned in clippy.toml) andunnecessary_get_then_check(applylive.rs:695). CI'svalidatejob, with clippy 1.99, passes.just test-tmt logically-bound-apply-live(ostree, centos-bootc stream10): fails at test-logically-bound-apply-live.nu:127, see the first inline comment. With the test image's three otherbound-images.dlinks removed inbuild_image, the plan passes, which confirms the cause.- The two fast-path repros and an extra
test_check_diff_scopecase (symlink in an added dir) were scratch changes, not pushed.
Generated-by: https://github.com/jmarrero-bot#llms
| .filter_map(|s| quadlet_relpath(&s.path)) | ||
| .collect(); | ||
| let live_quadlets: BTreeSet<&Utf8Path> = live.iter().map(|q| q.relpath.as_path()).collect(); | ||
| if live_quadlets == expected_quadlets && live.iter().all(|q| q.checksum == staged_csum) { |
There was a problem hiding this comment.
expected_quadlets counts every bound quadlet in the staged deployment, but only changed ones ever get a /run override. So whenever an unchanged bound quadlet is present, this condition is never true. That is the case for the test image: hack/provision-configure.sh binds curl.container, curl-base.image and podman.image. As a result the PR's own booted test fails at this head (test-logically-bound-apply-live.nu:127). After --no-restart, the plain re-run takes the full path and prints updated: ... / Applied 1 bound image definition change(s) instead of restart pending: lbi-sleeper.service. The already applied check after it would fail for the same reason.
The opposite case is also wrong. If the staged deployment has no bound quadlets at all and nothing is live yet, both sets are empty and all() is vacuously true. Then this returns "already applied" with exit 0, and:
- removing the last bound
.containernever stops its unit; - the deployment's pull-only bound images are neither pulled nor added to
protected-images.list; check_diff_scopeis skipped, so arbitrary out-of-scope changes report success.
Reproduced in a scratch extension of the booted test, with the test image's other bound images dropped. A staged image that only removes lbi-sleeper.container (and its bound-images.d link) gives Bound images from staged deployment are already applied, exit 0, and lbi-sleeper.service stays active.
Suggestion: drop the fast path, or derive it from compute_changes (booted specs are cheap to query) and require it to be non-empty, so it only fires when every change has a live marker for this checksum and nothing is left to stop.
| return Ok(()); | ||
| } | ||
| systemctl(&["daemon-reload"])?; | ||
| return restart_pending(rootfs, &live, sepolicy.as_ref()); |
There was a problem hiding this comment.
This path only restarts units whose marker has pending=true. Stops are never persisted, so they are lost on a re-run: the old_unit of a ServiceName= rename, the units of Removed changes, and stale overrides. Example: booted foo.container (unit foo.service), staged sets ServiceName=bar. apply-live --no-restart writes the marker for bar.service. A plain re-run then matches the fast path, restarts bar.service, and leaves foo.service running next to it. The same thing happens if the first run's systemctl stop foo.service fails. Reproduced with that same setup and ServiceName=lbi-renamed: after --no-restart and then a plain run (restart pending: lbi-renamed.service), both lbi-sleeper.service and lbi-renamed.service are active.
So the module doc's "a failure part way through is safe to resume from" (and the docs' "re-running after a failed restart") only holds for restarts.
One option: record pending stops in the state (e.g. a stop=foo.service key in the replacement's marker, plus a small list under /run/bootc/apply-live/ for pure removals and stale reverts), and clear each entry only after its stop succeeds.
| } else if ft.is_file() { | ||
| out.push(path); | ||
| } | ||
| // Symlinks (e.g. a unit masked via `-> /dev/null`) and other |
There was a problem hiding this comment.
walk_files is shared with check_diff_scope, which uses it to list the contents of added/removed directories, because the ostree diff doesn't enumerate them. Skipping symlinks here means a symlinked file inside an added in-scope dir is no longer reported as out of scope. For example, a new /usr/etc/containers/systemd/ that also carries an unbound other.container -> ... passes the check, and apply-live then reports success for a deployment it didn't fully apply. A test case added to test_check_diff_scope (symlink in the added dir) fails at this head: check_diff_scope returns Ok(()).
Suggest keeping walk_files reporting every non-directory entry, and doing the regular-file filtering in scan_live_quadlets only (e.g. symlink_metadata before reading).
| .get(quadlet) | ||
| .is_some_and(|prev| !prev.pending && prev.contents == c.spec.contents); | ||
| // Refuse to clobber a `/run` quadlet we didn't write. | ||
| if live_by_path.get(quadlet).is_none() && rootfs.try_exists(&dest)? { |
There was a problem hiding this comment.
This check runs inside the materialize loop, after pull_images, write_protected_images and any earlier writes. If it fails for the second quadlet, the first one is already in /run (pending, no daemon-reload), and any unrelated systemctl daemon-reload will pick it up. Copilot's suggestion was to check destination ownership before making changes: suggest hoisting this check into a pass over changes before the pull. (Also, clippy 1.92 with the Makefile flags flags get(quadlet).is_none() here as unnecessary_get_then_check. !live_by_path.contains_key(quadlet) reads better.)
| pub(crate) fn status_from_host() -> Result<Option<LiveBoundImages>> { | ||
| let rootfs = Dir::open_ambient_dir("/", cap_std::ambient_authority()).context("Opening /")?; | ||
| let live = scan_live_quadlets(&rootfs)?; | ||
| let Some(checksum) = live.first().map(|q| q.checksum.clone()) else { |
There was a problem hiding this comment.
If an apply of B over a live A fails partway (for example the refusal above), the markers carry two checksums. This takes whichever file the directory scan returns first and attributes every image to it, and status.rs then resolves a deployment from it. Minor, but it would be more accurate to expose the checksum per image, or to return None/an error when the markers disagree.
| Because `/run` is transient, the override disappears on the next boot, at | ||
| which point the staged deployment's own content applies. (A soft reboot | ||
| preserves `/run`; the override is cleared when one is prepared.) The applied | ||
| state is shown in `bootc status` and recorded in `/run/bootc/apply-live/`; |
There was a problem hiding this comment.
This is stale after bd5345b: the state now lives in the /run/containers/systemd/ markers, and /run/bootc/apply-live/ holds only the GC protection list.
Add new code to solve problems that was found in review by CoPilot and other AI review tools. Assisted-by: AI
|
Addressed the review feedback with those changes:
|
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects can skip container removals and deployment-scope validation, overwrite administrator overrides, and clear image protection while containers are still running.
11 open issues
Respect quadlet search precedence before materializing vendor definitions · New Check ownership before deleting removed quadlet overrides · New Require recorded completion for removal-only and pull-only updates · New Run the commit scope check before the fast path · New Preserve live overrides and image protection until soft-reboot handoff · New Read live-bound image state from bootc status JSON Reject local Quadlet conflicts before writing vendor definitions Stop previous service when effective unit identity changes Stop the previous live unit when renaming or removing a quadlet · New Mark materialized units for restart when planned stops include them · New Start the restored booted unit after reverting a renamed override · New
6 resolved since last review
Persist pending stops and reverts across stale override removal Preserve replacement destinations when moving bound Quadlets Reconcile complete Quadlet changes before restart-only path Retain protected image references until units stop using them Refresh already-live marker to the current staged checksum Skip non-regular entries before reading bootc markers
Review effort: Balanced
| let quadlet = quadlet_relpath(&spec.path).map(ToOwned::to_owned); | ||
| let unit = unit_for(quadlet.as_deref(), &spec.contents)?; |
| if !matches!(c.kind, ChangeKind::Added | ChangeKind::Updated) { | ||
| continue; | ||
| } | ||
| let Some(quadlet) = c.quadlet.as_deref() else { | ||
| continue; | ||
| }; |
| let materialize_done = stale.is_empty() | ||
| && materialization_complete(&changes, &live_by_path, staged_csum.as_str(), rootfs)?; |
| // `compute_changes` so unchanged bound quadlets (which never get a | ||
| // `/run` override) don't block it, and an empty change set with | ||
| // nothing pending doesn't spuriously report "already applied". | ||
| if materialize_done { |
| // A soft reboot preserves /run, so anything applied live must not leak | ||
| // into the target deployment. | ||
| crate::applylive::clear_state_from_host()?; |
| let quadlet_run_dir = Utf8Path::new(QUADLET_RUN_DIR); | ||
| let filemode = rustix::fs::Mode::from_raw_mode(0o644); | ||
| let mut written: BTreeMap<&Utf8Path, bool> = BTreeMap::new(); | ||
| let mut stops = pending_stops; |
| .is_some_and(|prev| !prev.pending && prev.contents == c.spec.contents); | ||
| // SAFETY: We know there's a parent | ||
| ensure_quadlet_dir(rootfs, dest.parent().unwrap(), sepolicy.as_ref())?; | ||
| let pending = !already_live && c.unit.is_some(); |
| let Some(unit) = q.unit.as_deref() else { | ||
| continue; | ||
| }; | ||
| let booted_has = booted_specs | ||
| .iter() | ||
| .any(|s| quadlet_relpath(&s.path) == Some(q.relpath.as_path())); | ||
| if booted_has { | ||
| systemctl(&["start", unit])?; | ||
| } |


Basic validation done here, draft while I review further and test manually.