Skip to content

feat(taskworker): Send heartbeats for running tasks - #859

Merged
evanh merged 6 commits into
mainfrom
evanh/feat/worker-heartbeat
Nov 17, 2026
Merged

evanh merged 6 commits into
mainfrom
evanh/feat/worker-heartbeat

Conversation

@evanh

@evanh evanh commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Add a new child process that is responsible for tracking heartbeat tasks. The parent starts a
heartbeat process that keeps track of all the activations that have a heartbeat set. It periodically
loops through those activations and calls the new update heartbeat endpoint on the broker.
When an activation is completed on the broker, it is removed from the tracked set of activations.

Both the parent and child keep a copy of the state of track activations, so if the child crashes the
parent can correctly restart it with all the tracked activations intact. Incoming/outgoing activations
update the state in the parent, and are synchronized to the child process using a channel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh
evanh requested a review from a team as a code owner November 10, 2026 19:37
@evanh

evanh commented Nov 10, 2026

Copy link
Copy Markdown
Member Author
$ cd clients/python && uv run pytest -q tests/worker/test_heartbeat.py tests/test_registry.py
35 passed in 0.56s

$ cd clients/python && uv run pytest -q
301 passed, 1 skipped
(a second run fails tests/test_metrics.py::test_track_memory_usage; it flakes the same way on main: "1 failed, 290 passed")

$ uv run mypy clients/python
Success: no issues found in 53 source files

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

"""Renew the heartbeat of every tracked activation once per interval."""
# The parent owns shutdown and tells us through shutdown_event.
signal.signal(signal.SIGINT, signal.SIG_IGN)
signal.signal(signal.SIGTERM, signal.SIG_IGN)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heartbeat process ignores terminate signals

Medium Severity

The heartbeat child ignores SIGTERM, but shutdown and multiprocessing daemon cleanup both use terminate() (SIGTERM). That fallback is a no-op, so a child stuck in import_app or a gRPC call is not killed. Python then join()s that daemon on interpreter exit, which can hang worker shutdown until the child happens to finish.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit afcfb0a. Configure here.

…on test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh

evanh commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

b7fa36a adds pull mode (heartbeats go to the broker host each task came from), starts the heartbeat process when the worker starts, and adds test_heartbeat to the integration tests and CI.

$ uv run python -m pytest integration_tests/integration_tests/test_heartbeat.py
1 passed in 19.67s

$ cd clients/python && uv run pytest -q
302 passed, 1 skipped, 1 failed (tests/test_metrics.py::test_track_memory_usage, flaky on main too)

$ uv run mypy clients/python
Success: no issues found in 53 source files

Manual run of the real heartbeat process (spawn, examples app, pull broker, 2s heartbeat):

after 8s: ('Processing', 0)
killed heartbeat process 11171
taskworker.worker.heartbeat.restarted
after kill + 6s: ('Processing', 0) new pid: 11219 alive: True
after untrack + 6s: ('Pending', 1)

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b7fa36a. Configure here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evanh and others added 3 commits November 16, 2026 15:18

@lvthanh03 lvthanh03 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for addressing the comment, looks good to me!

@evanh
evanh merged commit c5dc3a5 into main Nov 17, 2026
29 checks passed
@evanh
evanh deleted the evanh/feat/worker-heartbeat branch November 17, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants