Skip to content

feat(taskworker): Serve push health checks on a separate port - #869

Merged
evanh merged 1 commit into
mainfrom
evanh/feat/worker-health-server
Nov 18, 2026
Merged

evanh merged 1 commit into
mainfrom
evanh/feat/worker-health-server

Conversation

@evanh

@evanh evanh commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

By limiting the maximum_concurrent_rpcs, this limits how many requests can be in the gRPC queue waiting to be processed. This ensures that the worker server returns RESOURCE_EXHAUSTED in more scenarios, instead of Envoy waiting for timeouts.

Also adds an option to put the worker's healthserver on a separate port from the main server. This means that health checks won't end up in the same gRPC queue as task requests. This requires changes to how the workers are configured to work (adding the --health-port) option.

Refs STREAM-2103

Saturated push workers queue PushTask RPCs in the grpc executor with no limit, so health checks wait behind
them, time out, and get the pod ejected. With --health-port set, health gets its own server and the push server
rejects RPCs beyond --concurrency with RESOURCE_EXHAUSTED instead of queueing them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh
evanh requested a review from a team as a code owner November 17, 2026 20:03
@evanh

evanh commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Checked the premise with real grpc servers (grpcio from sentry's venv): 2 handler threads, 6 concurrent RPCs that each block 5s (standing in for PushTask on a full child queue), then one Health/Check with a 5s deadline.

today (no limit, shared)         health=DEADLINE_EXCEEDED in 5.01s  pushes={'OK': 6}
limit only (shared health)       health=RESOURCE_EXHAUSTED in 0.00s  pushes={'RESOURCE_EXHAUSTED': 4, 'OK': 2}
limit + separate health          health=SERVING in 0.00s  pushes={'RESOURCE_EXHAUSTED': 4, 'OK': 2}

maximum_concurrent_rpcs is server-wide (grpc/_server.py active_rpc_count) and counts executor-queued calls, so it only works with health moved off the push server.

uv run pytest -q                      # 311 passed, 1 failed: test_track_memory_usage (flaky, passes alone)
uv run --project clients/python mypy worker.py cli.py test_worker.py   # Success: no issues found

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 83e8a31. Configure here.

@linear-code

linear-code Bot commented Nov 17, 2026

Copy link
Copy Markdown

STREAM-2103

@evanh
evanh merged commit 4b02701 into main Nov 18, 2026
29 checks passed
@evanh
evanh deleted the evanh/feat/worker-health-server branch November 18, 2026 18:38
evanh added a commit to getsentry/sentry that referenced this pull request Nov 18, 2026
Passes through to `PushTaskWorker` from getsentry/taskbroker#869. Needs
a taskbroker-client bump to the release containing that change before
this can merge.

Refs STREAM-2103
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants