Skip to content

fix(api): track background goroutines so shutdown drains in-flight events - #3375

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/issue-3357-track-background-goroutines
Open

fix(api): track background goroutines so shutdown drains in-flight events#3375
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/issue-3357-track-background-goroutines

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #3357.

The API shutdown path waited for HTTP/gRPC handlers to return but silently abandoned background goroutines spawned during request handling (analytics events, volume cleanup). This PR introduces a sync.WaitGroup on Orchestrator to track those goroutines and waits for them with a 30-second budget after the HTTP drain completes.

What is tracked

Goroutine Site Why it matters
handleNewlyCreatedSandbox sandbox/store.go analytics insert + posthog event per sandbox created
analyticsRemove (x2) orchestrator/delete_instance.go billing-critical analytics event per sandbox deleted
removeSandboxFromNode cleanup orchestrator/create_instance.go kills orphaned sandboxes on store-add failure
deleteVolume handlers/volume_delete.go, volume_create.go removes volume data from orchestrator node

What is intentionally NOT tracked

  • BuildStatusSync in template-manager/create_template.go — can run up to 1 hour; cancellation via context is the correct shutdown signal for long-running polling loops.
  • Lifecycle goroutines (keepInSync, startStatusLogging, etc.) — managed by the root context and terminate when it is cancelled.

New API surface

// Orchestrator
func (o *Orchestrator) GoBackground(fn func())              // replaces bare "go"
func (o *Orchestrator) WaitBackground(ctx context.Context) bool

// APIStore (delegates to orchestrator)
func (a *APIStore) GoBackground(fn func())
func (a *APIStore) WaitBackground(ctx context.Context) bool

Shutdown sequence (after)

signal -> 15s health drain -> HTTP+gRPC parallel drain (up to 75s)
       -> background goroutine drain (up to 30s, new)
       -> cleanup()

Test plan

  • 4 new unit tests in orchestrator/background_wg_test.go:
    • TestGoBackgroundTracksGoroutine — goroutine is awaited
    • TestWaitBackgroundTimesOutOnSlow — ctx expiry returns false
    • TestGoBackgroundMultipleGoroutines — N concurrent goroutines all drained
    • TestWaitBackgroundNoGoroutinesReturnsImmediately — zero-goroutine fast path
  • go build ./packages/api/... clean
  • All existing ./packages/api/... tests pass

/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi Looking forward to your code review.

…ents

Add a sync.WaitGroup (bgWg) to Orchestrator to track short-lived background
goroutines spawned during request handling that must complete before process
exit to avoid dropping analytics events or volume cleanup.

Tracked goroutines:
- handleNewlyCreatedSandbox (analytics insert + posthog, via sandbox store)
- analyticsRemove (billing-critical analytics, 2 sites in delete_instance.go)
- removeSandboxFromNode error-path cleanup (create_instance.go)
- deleteVolume background goroutines (volume_delete.go, volume_create.go)

New helper methods:
- Orchestrator.GoBackground(fn) — wraps bare "go" with Add/Done
- Orchestrator.WaitBackground(ctx) — context-bounded drain
- APIStore.GoBackground / WaitBackground — thin delegates for handler use

main.go: after the HTTP/gRPC drain completes, wait up to bgShutdownTimeout
(30s) for background goroutines before calling cleanup(). Removes the
long-standing TODO comment that described exactly this gap.

Resolves e2b-dev#3357
@AdaAibaby
AdaAibaby force-pushed the fix/issue-3357-track-background-goroutines branch from 4e7073e to 53cea2e Compare July 24, 2026 04:55
@ValentaTomas
ValentaTomas force-pushed the main branch 2 times, most recently from 5aad415 to d71980e Compare July 25, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(api): background goroutines (events, builds) not awaited at graceful shutdown — events may be dropped

2 participants