Skip to content

fix(worker): Report task failures inside the task's Sentry scope - #868

Draft
dashed wants to merge 1 commit into
mainfrom
aleal/report-task-failure-in-task-scope
Draft

dashed wants to merge 1 commit into
mainfrom
aleal/report-task-failure-in-task-scope

Conversation

@dashed

@dashed dashed commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Bottom line: The worker now reports a task failure inside the task's own Sentry isolation scope. Failure events then carry the task's tags, contexts and trace, in every app that uses this client.

  • Why it matters: Today a failure event has none of the task's scope data. For example, a getsentry billing task failure event had no organization; we had to search the logs for it.
  • What it does not do: It does not change log messages, extra fields, fingerprints, transaction names (fix(worker): Attribute captured task errors to the current task #818), retry decisions, task status or metrics. Scope data does not reach the next task.
  • Behavior change to review: Failure events now carry the task's tags, contexts, user, breadcrumbs and trace id. A level that the task sets on its scope (for example with sentry_sdk.set_level()) now applies to all of its failure events. A fingerprint that the task sets applies to the ordinary failure event only; the timeout and no-retries reports keep their own. The taskworker.task.failed, .retry_exhausted and .retry records in Sentry Logs now carry the task's trace id and scope attributes.

Cause

_execute_activation opens the task's isolation scope (workerchild.py:670). set_tag() and set_context() in the task write to it. A raised exception leaves that with block, so the SDK restores the worker scope. Then run_worker reports the timeout (:511-522), no retries remaining (:553-573) or other failure (:583-587) in a fork of the worker scope. The task's data and trace are gone.

Fix

run_worker opens the task's isolation scope around the task call and the except blocks. _execute_activation does not open one. The nested isolation_scope() blocks in the reports stay and now fork the task scope. Review with whitespace hidden: the try block moves one indent level.

The retry log is in the same except block, so it also runs in the task scope. It is an info log and makes no error event.

Tests

The new example task examples.tagged sets a tag and a context, then passes, raises or sleeps.

  • test_child_process_error_event_has_task_scope_data (exception, retry-exhausted, timeout): the event carries the task's tag, context and trace id. Red on main, green here.
  • test_child_process_task_scope_data_does_not_reach_next_task (after-success, after-failure): the next task's event has none of the first task's data. Passes on main and here; fails with no task scope at all.
  • Client suite: 315 passed, 1 skipped (main: 310). pre-commit (black, isort, flake8, mypy) passes.
Test output
# main
FAILED test_child_process_error_event_has_task_scope_data[exception] - KeyError: 'tags'
FAILED test_child_process_error_event_has_task_scope_data[retry-exhausted] - KeyError: 'tags'
FAILED test_child_process_error_event_has_task_scope_data[timeout] - KeyError: 'tags'
PASSED test_child_process_task_scope_data_does_not_reach_next_task[after-success]
PASSED test_child_process_task_scope_data_does_not_reach_next_task[after-failure]
3 failed, 2 passed

# this branch
5 passed

# no task scope at all (guard check)
FAILED test_child_process_task_scope_data_does_not_reach_next_task[after-success]
  AssertionError: assert 'example.task_tag' not in {'example.task_tag': 'from-first-task'}
FAILED test_child_process_task_scope_data_does_not_reach_next_task[after-failure]
  AssertionError: assert 'example.task_tag' not in {'example.task_tag': 'from-first-task'}

A task runs in its own Sentry isolation scope, opened in
_execute_activation. The failure reports in run_worker ran after that
scope had closed. So the tags, contexts and trace that a task set did
not reach the task's error events.

Open the task's isolation scope in run_worker, around the task call and
the failure reports. Each task still gets a fresh fork of the worker
scope, so nothing reaches the next task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant