Repository navigation
feat(taskworker): Serve push health checks on a separate port - #869
Merged
Merged
Conversation
Saturated push workers queue PushTask RPCs in the grpc executor with no limit, so health checks wait behind them, time out, and get the pod ejected. With --health-port set, health gets its own server and the push server rejects RPCs beyond --concurrency with RESOURCE_EXHAUSTED instead of queueing them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Checked the premise with real grpc servers (grpcio from sentry's venv): 2 handler threads, 6 concurrent RPCs that each block 5s (standing in for PushTask on a full child queue), then one Health/Check with a 5s deadline.
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 83e8a31. Configure here.
enochtangg
approved these changes
Nov 17, 2026
3 tasks done
evanh
added a commit
to getsentry/sentry
that referenced
this pull request
Nov 18, 2026
Passes through to `PushTaskWorker` from getsentry/taskbroker#869. Needs a taskbroker-client bump to the release containing that change before this can merge. Refs STREAM-2103
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

By limiting the
maximum_concurrent_rpcs, this limits how many requests can be in the gRPC queue waiting to be processed. This ensures that the worker server returns RESOURCE_EXHAUSTED in more scenarios, instead of Envoy waiting for timeouts.Also adds an option to put the worker's healthserver on a separate port from the main server. This means that health checks won't end up in the same gRPC queue as task requests. This requires changes to how the workers are configured to work (adding the --health-port) option.
Refs STREAM-2103