fix(api): track background goroutines so shutdown drains in-flight events - #3375
Open
AdaAibaby wants to merge 1 commit into
Open
fix(api): track background goroutines so shutdown drains in-flight events#3375AdaAibaby wants to merge 1 commit into
AdaAibaby wants to merge 1 commit into
Conversation
…ents Add a sync.WaitGroup (bgWg) to Orchestrator to track short-lived background goroutines spawned during request handling that must complete before process exit to avoid dropping analytics events or volume cleanup. Tracked goroutines: - handleNewlyCreatedSandbox (analytics insert + posthog, via sandbox store) - analyticsRemove (billing-critical analytics, 2 sites in delete_instance.go) - removeSandboxFromNode error-path cleanup (create_instance.go) - deleteVolume background goroutines (volume_delete.go, volume_create.go) New helper methods: - Orchestrator.GoBackground(fn) — wraps bare "go" with Add/Done - Orchestrator.WaitBackground(ctx) — context-bounded drain - APIStore.GoBackground / WaitBackground — thin delegates for handler use main.go: after the HTTP/gRPC drain completes, wait up to bgShutdownTimeout (30s) for background goroutines before calling cleanup(). Removes the long-standing TODO comment that described exactly this gap. Resolves e2b-dev#3357
AdaAibaby
force-pushed
the
fix/issue-3357-track-background-goroutines
branch
from
July 24, 2026 04:55
4e7073e to
53cea2e
Compare
ValentaTomas
force-pushed
the
main
branch
2 times, most recently
from
July 25, 2026 22:53
5aad415 to
d71980e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3357.
The API shutdown path waited for HTTP/gRPC handlers to return but silently abandoned background goroutines spawned during request handling (analytics events, volume cleanup). This PR introduces a
sync.WaitGrouponOrchestratorto track those goroutines and waits for them with a 30-second budget after the HTTP drain completes.What is tracked
handleNewlyCreatedSandboxsandbox/store.goanalyticsRemove(x2)orchestrator/delete_instance.goremoveSandboxFromNodecleanuporchestrator/create_instance.godeleteVolumehandlers/volume_delete.go,volume_create.goWhat is intentionally NOT tracked
BuildStatusSyncintemplate-manager/create_template.go— can run up to 1 hour; cancellation via context is the correct shutdown signal for long-running polling loops.keepInSync,startStatusLogging, etc.) — managed by the root context and terminate when it is cancelled.New API surface
Shutdown sequence (after)
Test plan
orchestrator/background_wg_test.go:TestGoBackgroundTracksGoroutine— goroutine is awaitedTestWaitBackgroundTimesOutOnSlow— ctx expiry returns falseTestGoBackgroundMultipleGoroutines— N concurrent goroutines all drainedTestWaitBackgroundNoGoroutinesReturnsImmediately— zero-goroutine fast pathgo build ./packages/api/...clean./packages/api/...tests pass/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi Looking forward to your code review.