feat(snapshots): in place resume & fs-only snapshots - #3483
Conversation
Adds the overlay primitives the in-place background rootfs seal needs, on top of main's #3320 deferred-export code: - Overlay: atomic writable-cache pointer + a "sealing" slot. SwapCache installs a fresh writable cache and moves the previous one to the sealing slot; ReadAt resolves writable -> sealing -> base. FoldSealing / ReleaseSealing collapse the sealing cache back once its background seal is done. - Cache.FillMissingFrom copies the sealing cache's blocks the writable cache lacks (concurrency-safe writeAtIfAbsent keeps the guest's newer writes), so the writable cache is a complete diff again after a fold. Cache.DiffMetadata already exists on main (from #3320) and is reused. Purely additive; overlay.go was untouched by #3320. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
Adds the NBD provider capabilities the in-place background seal needs, alongside main's #3320 PrepareExportDiff (destroy-path eject+stop): - ExportDiffInPlace: reflink the overlay's dirty blocks without ejecting the cache, so the sandbox keeps running on it (synchronous in-place fallback). - SwapForBackgroundSeal: flush the device, swap a fresh writable cache onto the live overlay, return the frozen previous cache for background sealing. - FoldSealed: fold the sealing cache back into the writable cache and detach it. DirectProvider stubs all three (NBD-only). Reuses main's ejectAndStopSandbox. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
Enable resuming the same Firecracker VM after a memory snapshot, with its memory backing intact: - Process.ResumeInPlace un-pauses the already-running FC process (no reconfigure, no UFFD handshake, no snapshot load) — the process/memory/rootfs are all live. - Memory.PeekMemfd borrows the memfd without consuming it (Uffd loads instead of swapping; NoopMemory returns nil). - block.NewCacheFromMemfdKeepOpen copies the dirty guest pages WITHOUT closing the memfd; ExportMemory + pauseProcessMemory + processMemorySnapshot gain a keepMemfdOpen flag that takes this path and disables dedup / inflight-serve / provisional serving (which would consume or double-manage the fd the live guest still faults on). keepMemfdOpen is threaded but always false so far (Pause passes false) — no behavior change until the in-place branch wires it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
…ckpoint Wire the in-place path on top of main's deferred-export machinery, gated by the existing defer-rootfs-export flag (no new flag): - pauseOptions gains maintainSandbox (WithMaintainSandbox). Pause registers a resume-on-error, snapshots memory with keepMemfdOpen, waits out any prior in-place seal, then after writing metadata ResumeInPlace + unfreeze/thaw + restart checks BEFORE starting the (possibly deferred) rootfs seal, so the reflink stays off the resume critical path. - processRootfsSnapshot gets a maintainSandbox branch: with defer-rootfs-export it swaps a fresh COW cache (SwapForBackgroundSeal) and runs runInPlaceRootfsExport in the background (reflink -> resolve deferred diff -> FoldSealed + release, reusing main's sealCacheToDiff); without the flag it exports in place synchronously (RootfsDiffCreator with nil closeHook -> ExportDiffInPlace). A provider that can't swap (DirectProvider -> ErrDeferredExportNotSupported) falls back to the synchronous in-place export. - waitForRootfsSeal serializes the next in-place checkpoint behind the prior seal's fold so the writable cache is a complete diff before it swaps again. - Checkpoint handler resumes in place: drop MarkStopping/stopSandboxAsync and the resume-fresh (GetTemplate/ResumeSandbox/lifecycle/upgrade/markLive) path; the original sandbox keeps running and the upload targets it. Prefetch is omitted (live tracker over-prefetches). snapshotAndCacheSandbox gains maintainSandbox. The base device stays on template T (no rebase), so the live sandbox never depends on the not-yet-durable checkpoint build. fs-only in-place checkpoint (proto/API) is a separate follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
Wire a filesystem_only flag through the Checkpoint gRPC handler. Unlike the earlier destroy-and-resume-fresh design, the in-place checkpoint needs no new mode: fs-only checkpoint is just Pause with WithFilesystemSnapshot() + WithMaintainSandbox() together, which the current Pause already handles — it fsfreeze-quiesces the guest, skips the memory snapshot (mem stays NoDiff, so no memfile is produced or uploaded), seals the rootfs in the background, resumes the same VM in place, and thaws it (bestEffortFsthaw is already gated on envd fsfreeze support, so no 404 teardown on old envd). The persisted template is marked filesystem-only and cold-boots (reboots) on launch; the live sandbox keeps its memory and running processes. Only the handler flag mapping changes (was hardcoded false); prefetch is already omitted for in-place checkpoints. No SnapshotMode enum, no build_upload guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
Add an optional `memory` boolean (default true) to SandboxSnapshotRequest. When false, PostSandboxesSandboxIDSnapshots derives FilesystemOnly and threads it through SnapshotTemplateOpts -> the upsert params and the orchestrator SandboxCheckpointRequest, producing a rootfs-only template whose sandboxes cold-boot (reboot) from disk. Default behavior (full memory snapshot) is unchanged. Regenerated api.gen.go and the integration client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
PR SummaryHigh Risk Overview Reviewed by Cursor Bugbot for commit da32f8a. Bugbot is set up for automated code reviews on this repo. Configure here. |
d4b28b7 to
20bbe96
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| s.bestEffortUnfreeze(ctx) | ||
| if pauseOpts.filesystemSnapshot { | ||
| s.bestEffortFsthaw(ctx) | ||
| } |
There was a problem hiding this comment.
In-place fs thaw skips exec path
High Severity
On a successful filesystem-only in-place checkpoint, only bestEffortFsthaw runs. That no-ops when envd lacks native /fsfreeze and the rootfs was frozen via the exec path. Cleanup thaw is error-only, so the live guest can stay FIFREEZE'd after resume.
Reviewed by Cursor Bugbot for commit 20bbe96. Configure here.
|
|
||
| return s.process.ResumeInPlace(ctx) | ||
| }) | ||
| } |
There was a problem hiding this comment.
Checks not restarted after error resume
Medium Severity
Checks.Stop() runs at the start of Pause, but NewChecks/Start only run on the in-place success path. If snapshotting fails after FC pause, cleanup resumes the VM via ResumeInPlace and leaves health checks permanently stopped.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 20bbe96. Configure here.
…ecks) Three fixes from the PR #3483 review of the in-place checkpoint path: - Memfd consumed before peek (High): processMemorySnapshot called Memfd() unconditionally, which swaps the memfd out of uffd, so the keepMemfdOpen branch's PeekMemfd returned nil (broke the in-place keep-open export and leaked/dropped the taken fd). Now the in-place path uses PeekMemfd (borrow) and only the destroy path consumes via Memfd. - In-place fs-thaw skips exec path (High): the success resume only ran the native bestEffortFsthaw, a no-op when the rootfs was frozen via the exec path (old envd) — leaving the live in-place guest's filesystem frozen. Now thaws to match guestPrepareFsForPause: native /fsthaw when supported, else exec fsfreeze -u. (The prior "frozen state discarded when the VM stops" assumption doesn't hold for in-place resume.) - Checks not restarted after error resume (Medium): the resumeOnError cleanup resumed the VM but left health checks (stopped at Pause start) off. It now restarts NewChecks/Start after ResumeInPlace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cefcc89. Configure here.
| SandboxId: sbx.SandboxID, | ||
| BuildId: upsertResult.BuildID.String(), | ||
| Metadata: map[string]string{storageopts.ObjectMetadataTemplateID: snapshotTemplateEnvID}, | ||
| FilesystemOnly: opts.FilesystemOnly, |
There was a problem hiding this comment.
Checkpoint errors kill live sandbox
High Severity
In-place checkpoint keeps the sandbox running when upload (or other post-resume steps) fail, but the API still RemoveSandboxs on Checkpoint errors. That kills a healthy resumed sandbox after a template persistence failure.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit cefcc89. Configure here.
1 participant
TBD