[vineethsai7]

Summary

Recover a stopped MicroVM sandbox whose writable overlay has run out of usable space when its saved files can be retained. If the driver cannot establish that, start fails with OverlayDiskFull and leaves the image untouched. Recovery runs in the VM driver's offline overlay preparation step before the next guest starts.

Related Issue

Closes #5149

The issue remains state:triage-needed; design feedback is welcome.

Changes

• Check saved block usage with read-only e2fsck, then perform preen, cleanup, and any growth on a private copy. Publish the copy only after it passes the checks; leave the original image untouched on refusal.
• Require a durable guest filesystem flush acknowledgement for new VM runs. Refuse an unacknowledged restart, or a legacy overlay with an unreplayed journal and no acknowledgement. Include the VZ runtime-initiated Unix listener in guest flushing.
• Refuse recovery if journal replay or repair releases saved blocks, preen requests interactive repair, or a previously published overlay is missing. Keep the existing file paths rather than reconnecting orphans into lost+found.
• Preserve a recovered larger overlay on later starts. Report growth through an OverlayDiskFull platform warning and use that reason for a refused start. Document the recovery path, host tools, and overlay_disk_mib behavior.

Design

Earlier full-overlay evidence captured a dirty image whose superblock claimed almost all blocks free while a read-only check counted almost no free blocks. The VZ guest transport omitted the filesystem flush and the old stop path edited dirty images with debugfs before journal replay. The image from the data-loss run was deleted, so these are confirmed unsafe paths rather than a definitive postmortem of that run. The driver now waits for a durable guest flush acknowledgement, refuses an unacknowledged dirty journal, and performs recovery on a copy.

The low-space decision uses checked ext4 free blocks minus reserved blocks. A preen pass that needs orphan reconnection or another interactive repair refuses startup, preserving the original image. resize2fs runs on the copy after a successful check. A read-only recovery boot would need a new guest and access lifecycle; the offline path keeps the CLI and RPC contracts unchanged.

User-facing example

The existing setting controls the initial size of new overlays. A recovery can grow an existing overlay beyond it:

[openshell.drivers.vm]
overlay_disk_mib = 1024

openshell sandbox stop full
openshell sandbox start full
openshell sandbox download full /sandbox/work ./work
openshell sandbox exec full -- rm -f /sandbox/fill

The watch stream emits an event like:

source: vm
type: Warning
reason: OverlayDiskFull
message: VM overlay was nearly full and grew by 64 MiB for recovery; remove files or increase [openshell.drivers.vm] overlay_disk_mib for new sandboxes

If recovery fails, the sandbox stays in Error with a Ready=False condition whose reason is OverlayDiskFull. Space and growth failures also name overlay_disk_mib.

Compatibility

New VM runs record a durable guest flush acknowledgement before an image may be recovered. An unacknowledged run, or a legacy dirty image without an acknowledgement, refuses start with OverlayDiskFull. Safe recovery checks a private copy of every preserved overlay; low-space recovery may add more than 64 MiB in one start, and later recoveries may grow the disk again, so the gateway host needs available storage. Changing overlay_disk_mib does not resize an existing overlay. Recovery needs resize2fs from e2fsprogs; gateway config preflight currently checks the other VM filesystem tools but not this one. Failed repair or growth retains the original overlay for inspection.

Release Notes

Stopped MicroVM sandboxes with full overlay disks can recover on start when their saved data is verifiable. OpenShell expands the overlay when needed and refuses to start with OverlayDiskFull if recovery could lose saved data.

Testing

• cargo fmt --all -- --check passed.
• cargo clippy -p openshell-driver-vm --all-targets -- -D warnings passed.
• Historical check at 3eeb96b0c: cargo test -p openshell-driver-vm --quiet passed (274 library, 23 binary, 5 integration). The later follow-up replaced automatic orphan repair with refusal and added the data-preservation regressions below.
• At 9618d3f467, documentation navigation check and Markdown lint passed. Fern check found 0 errors and 3 warnings; the touched page's MDX warning also occurs without that change, and the other warnings concern redirect authentication and theme contrast. These doc checks were not rerun on the later commits.
• At bc221364f008cb9e410f86fbe06d376c0ec4e082, cargo fmt --all -- --check and cargo clippy -p openshell-driver-vm -p openshell-sandbox -p openshell-supervisor --all-targets -- -D warnings passed through dcb. Driver tests passed (278 library, 23 binary, 5 integration), supervisor tests passed (264 library, 18 binary), and all 286 sandbox library tests passed with one test thread. A parallel sandbox run hit an unrelated existing file-descriptor race before the serial pass.
• Manual MicroVM proof at 3eeb96b0ce failed: one fresh sandbox reached Ready on the first restart after losing both saved files. Two other first restarts recovered. The lead must rerun stop/start/download on a real macOS host against bc221364f008cb9e410f86fbe06d376c0ec4e082; the build box has no KVM, Docker, or mise.

Follow-ups

• Report overlay usage before it reaches the recovery threshold.
• Add resize2fs to VM host-tool preflight with a supported probe; resize2fs does not accept the -V flag used for the current tools.
• Run the full stop, restart, and download scenario on macOS and Linux KVM hosts.

Checklist

• Follows Conventional Commits.
• Commits are signed off (DCO).
• Architecture and user documentation updated where applicable.

Review follow-up

• P1: Preserve overlays smaller than a new configured default — Valid; fixed in 9618d3f4677190ba59b1f1bb13179effd667517f. Preserved overlays are checked independently of the current overlay_disk_mib; the regression restarts a 64 MiB saved overlay after raising the default to 128 MiB and verifies its contents.
• P1: Refuse automatic repair when other corruption is present — Valid; initially fixed in 9618d3f4677190ba59b1f1bb13179effd667517f and tightened in bc221364f008cb9e410f86fbe06d376c0ec4e082. Recovery no longer retries orphan reconnection at all; a required interactive repair refuses startup without changing the original image. The regression covers an unattached inode and an additional unsafe repair request.
• First-restart data loss at 3eeb96b0ceaa2bf5c6e4d02382ef9b8d65d8b0ee — Valid; fixed by bc221364f008cb9e410f86fbe06d376c0ec4e082 and verified by the later macOS proof (3/3 first restarts). An earlier manual run at 3eeb96b0ceaa2bf5c6e4d02382ef9b8d65d8b0ee reached Ready after losing both saved files in one fresh sandbox. The lost image was deleted, so its exact journal outcome cannot be reconstructed. The failed run did not grow the image. Code audit found that flush failures were acknowledged, stop edited dirty images before journal replay, and a missing saved overlay could be recreated. The VM guest uses a runtime-initiated vsock listener even when the VZ host transport is Unix; VM flushing remains enabled. The follow-up records a durable acknowledgement, refuses an unacknowledged or legacy dirty image, repairs and grows only a private copy, rejects repair that releases saved blocks, defers guest cleanup until after replay, and refuses to replace a missing published overlay. The regression writes a known marker and filler until guest-usable ext4 blocks are exhausted, then checks that restart either retains both files or refuses without changing the image.
• P1: Do not classify runtime-initiated Unix as VM-only — Valid; fixed in 2787e6b10b4f87397bad647e4d8f57344730d275. VM boundary provisioning uses vsock listeners, including when its host transport is Unix. Docker and Podman use RuntimeInitiatedUnix; that listener no longer performs the VM filesystem flush. A regression terminates a runtime-initiated Unix container boundary without installing a flush and checks that VM vsock listeners still flush.
• P1: Roll back the run marker when launcher spawn fails — Valid; fixed in 2787e6b10b4f87397bad647e4d8f57344730d275. The driver durably restores the previous run and flush acknowledgement state if launcher spawn fails before a VM exists. A regression covers both an acknowledged saved overlay and a first launch, and checks that saved bytes remain unchanged.
• P2: Persist a successful flush on proxy failure — Valid; fixed in 2787e6b10b4f87397bad647e4d8f57344730d275. The supervisor writes the flush acknowledgement immediately after a successful guest termination, including proxy exit before or after the canonical process ends. A regression checks that proxy failure retains a successful acknowledgement and a failed termination cannot create one.
• P2: Publish the overlay before committing its marker — Valid; fixed in 2787e6b10b4f87397bad647e4d8f57344730d275. The complete staged overlay is renamed and its directory synced before overlay-published is written. A regression forces marker creation to fail and verifies that restore keeps and reads the published image.
• No review finding was judged not applicable.
• At 2787e6b10b4f87397bad647e4d8f57344730d275, cargo fmt --all -- --check and cargo clippy -p openshell-driver-vm -p openshell-sandbox -p openshell-supervisor --all-targets -- -D warnings passed through dcb. Driver and supervisor tests passed (cargo test -p openshell-driver-vm -p openshell-supervisor --quiet: 280 driver library, 23 driver binary, 5 driver integration, 265 supervisor library, 18 supervisor binary). Sandbox tests passed (cargo test -p openshell-sandbox --lib --quiet -- --test-threads=1: 287 library; cargo test -p openshell-sandbox --test stdout_logging --quiet -- --test-threads=1: 1 integration). Each of the four new regressions failed under a controlled pre-fix behavior and passed on the fixed source.
• Manual macOS MicroVM stop/start/download proof passed 3/3 first restarts at bc221364f008cb9e410f86fbe06d376c0ec4e082. The lead will rerun it at 2787e6b10b4f87397bad647e4d8f57344730d275 on a real host; the build box has no KVM, Docker, or mise.

Manual verification

PASS on Apple silicon Hypervisor.framework at PR head 2787e6b10b4f87397bad647e4d8f57344730d275. Three independent, fresh sandboxes with a 100%-full overlay each reached Ready on the first restart. In every case, two files saved before the disk filled downloaded with their exact contents, the filler file retained its size, and WatchSandbox reported a Warning event with reason OverlayDiskFull. Upstream main be5e4949976e47b49de78ebef770735e198f8b06 failed the same first restart in two fresh sandboxes; the saved file in each was unavailable for download.

Environment and method

The test host ran macOS 15.8 on arm64 with Hypervisor.framework. I built the PR and main snapshots in separate disposable local accounts with separate Cargo targets, using Rust/Cargo 1.95.0, Zig 0.14.1, cargo-zigbuild 0.22.3, CARGO_BUILD_JOBS=2, and nice +15. The PR lane cross-built its guest sandbox binary from the exact PR head. The main lane embedded the published v0.1.6 arm64 Linux guest binary (release checksum verified); each lane built its own host supervisor, VM init, CLI, gateway, and VM driver from its stated source commit. Both drivers embedded the same published darwin-aarch64 VM runtime and were signed ad hoc with the Hypervisor entitlement. Each gateway used separate private state, a private HOME, and its own loopback port. The overlay setting was overlay_disk_mib = 1024, with 2 vCPUs, 1024 MiB RAM, and ubuntu:24.04.

Build commands, run from the corresponding source snapshot inside a nice +15 script with private CARGO_TARGET_DIR and OPENSHELL_VM_RUNTIME_COMPRESSED_DIR:

ulimit -n 8192
# PR head: build its guest, supervisor, VM init, and host binaries.
cargo zigbuild --release -p openshell-sandbox --target aarch64-unknown-linux-musl
cargo build --release -p openshell-supervisor
cargo zigbuild --release -p openshell-driver-vm --bin openshell-vm-init --no-default-features --target aarch64-unknown-linux-musl
cargo build --release -p openshell-cli -p openshell-gateway -p openshell-driver-vm --features openshell-server/prebuilt-z3

# Main: embed the checksum-verified published v0.1.6 guest, then run:
cargo build --release -p openshell-supervisor
cargo zigbuild --release -p openshell-driver-vm --bin openshell-vm-init --no-default-features --target aarch64-unknown-linux-musl
cargo build --release -p openshell-cli -p openshell-gateway -p openshell-driver-vm --features openshell-server/prebuilt-z3

For each fresh PR sandbox I set GW to the private loopback endpoint, SB to a unique redacted name, and MARKER and SECOND to two distinct strings. The main baseline used the same flow with only recover.txt. The download attempts followed the first start, without a second stop/start:

openshell --gateway-endpoint "$GW" sandbox create --name "$SB" --keep --detach --from ubuntu:24.04 -- sleep infinity
openshell --gateway-endpoint "$GW" sandbox exec "$SB" -- sh -c "printf '%s\n' '$MARKER' > /sandbox/recover.txt; printf '%s\n' '$SECOND' > /sandbox/second.txt"
openshell --gateway-endpoint "$GW" sandbox exec "$SB" -- sh -c 'yes AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > /sandbox/fill; echo yes_rc=$?; df -h /; stat -c "fill_bytes=%s" /sandbox/fill'
openshell --gateway-endpoint "$GW" sandbox stop "$SB"
# Record the stopped overlay's byte size and SHA-256.
openshell --gateway-endpoint "$GW" sandbox start "$SB"
openshell --gateway-endpoint "$GW" sandbox get "$SB"
openshell --gateway-endpoint "$GW" sandbox download "$SB" /sandbox/recover.txt ./recover-downloaded.txt
openshell --gateway-endpoint "$GW" sandbox download "$SB" /sandbox/second.txt ./second-downloaded.txt
openshell --gateway-endpoint "$GW" sandbox exec "$SB" -- sh -c 'cat /sandbox/recover.txt /sandbox/second.txt; stat -c "fill_bytes=%s" /sandbox/fill; df -h /'

An initial baseline build shared the PR Cargo target and was discarded because it could reuse PR artifacts. The main results below are from the separate clean-target build. In all five reported full-overlay runs, yes ended with “No space left on device” and guest df reported 974M total, 958M used, 0 available, 100% on /. The PR filler was 1,003,548,672 bytes in each run. The baseline filler was 1,003,552,768 bytes in each run.

Before: main be5e49499

Both fresh baseline sandboxes failed their first start with ProcessExited: VM process exited with status 0 while waiting for Ready. Both entered Error and their immediate downloads returned “sandbox is not ready.” After stop, each second start failed with ProvisioningFailed: e2fsck 1.47.4 exited 4 while recovering the journal, reported unattached inode 24, and required manual fsck. Downloads still returned “sandbox is not ready.” This reproduced both failure stages from the issue.

# After the first baseline start and download fail:
openshell --gateway-endpoint "$GW" sandbox stop "$SB"
openshell --gateway-endpoint "$GW" sandbox start "$SB"
openshell --gateway-endpoint "$GW" sandbox download "$SB" /sandbox/recover.txt ./recover-second-attempt.txt

After: PR head 2787e6b10b

Fresh sandbox	First start	Immediate downloads	Files after restart	Overlay bytes before → after	WatchSandbox event
PR A	Ready	Both succeeded	Both markers matched; filler 1,003,548,672 bytes	1,073,741,824 → 1,161,822,208	Warning: OverlayDiskFull
PR B	Ready	Both succeeded	Both markers matched; filler 1,003,548,672 bytes	1,073,741,824 → 1,161,822,208	Warning: OverlayDiskFull
PR C	Ready	Both succeeded	Both markers matched; filler 1,003,548,672 bytes	1,073,741,824 → 1,161,822,208	Warning: OverlayDiskFull

The overlay grew by 84 MiB on each first restart. Guest df then showed 27 MiB available and 98% use. The WatchSandbox stream was read with follow_events=true and event_tail=20; each event named OverlayDiskFull and directed the user to remove files or increase [openshell.drivers.vm] overlay_disk_mib for new sandboxes. No sandbox was Ready with missing saved data, and no second start was needed to retrieve it.

For PR A, the measurement script initially exited after stop because macOS Bash lacks mapfile. That script had issued no start and made no overlay edit. The corrected script performed PR A's first start and all checks above.

Other corruption

On a separate disposable PR sandbox, I stopped a normal overlay and changed only its two-byte primary ext4 superblock magic at offset 1024 + 0x38. Read-only e2fsck -fn reported “Bad magic number in super-block.” The PR driver refused start with ProvisioningFailed (“overlay ... is not ext4”), and the sandbox entered Error. The damaged overlay's SHA-256 was identical before and after the read-only fsck and after the driver start attempt. This control covers that superblock corruption, not every ext4 failure mode.

Cleanup and scope

All eight disposable sandboxes, including the two discarded baseline trials, were deleted. All three private gateways and drivers exited, their loopback ports were free, and the disposable accounts, HOMEs, signing keys, source snapshots, and build/state directories were removed. The pre-existing standard gateway and driver retained their original PIDs. macOS left two empty, 0-byte, OS-protected per-user cache directories. This manual proof covers macOS arm64 Hypervisor.framework; Linux KVM, the container shutdown path, and injected launcher-spawn or crash windows were not separately run.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T03:58:15.062820Z	bc22136	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d40301b9f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bc221364f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".