Skip to content

Exit the pool worker after a task raises SystemExit or KeyboardInterrupt - #457

Merged
auvipy merged 1 commit into
celery:mainfrom
GangEunzzang:fix/worker-exit-after-systemexit
Sep 9, 2026
Merged

Exit the pool worker after a task raises SystemExit or KeyboardInterrupt#457
auvipy merged 1 commit into
celery:mainfrom
GangEunzzang:fix/worker-exit-after-systemexit

Conversation

@GangEunzzang

Copy link
Copy Markdown
Contributor

Summary

Since bd2f803 (#452) the worker loop catches BaseException around the task call so that a task raising e.g. BaseException is reported to the caller instead of surfacing as WorkerLostError. That also catches the SystemExit raised by billiard's own SIGTERM handler, which changes what happens on a hard time limit:

  1. on_hard_timeout()_trywaitkill() sends SIGTERM to the worker; the handler installed by reset_signals() raises SystemExit inside the running task.
  2. The worker now treats that as a task failure: it marshals the result, sends READY, and goes back to wait_for_job().
  3. A blocking-pool worker waits inside SimpleQueue.get_payload(), i.e. with self._rlock: return self._reader.recv_bytes() — holding the queue's shared read lock.
  4. _trywaitkill() follows up with SIGKILL 0.1 s later, so the worker dies while holding _rlock.
  5. The replacement worker blocks in _rlock.__enter__ forever and the pool never processes another job. Pool.terminate() blocks on the same lock (_help_stuff_finish() acquires inqueue._rlock), so the pool can't be shut down cleanly either.

Before #452 the SystemExit propagated out of workloop(), the worker exited through _do_exit() without touching the queue, and the SIGKILL was harmless.

This PR keeps the #452 behaviour (the exception is still reported to the caller) but exits the worker afterwards when the exception is SystemExit or KeyboardInterrupt, since either means the process was asked to go away. The pool then replaces the worker as it did before #452.

How it shows up

With Celery this is a prefork worker (blocking pool, e.g. the filesystem:// transport) that stops executing tasks after the first hard time_limit kill: the replacement child starts, the next task is "received", and nothing runs until the worker is restarted. On macOS, where billiard 4.3 defaults to spawn, the stuck replacement also survives the parent's shutdown as an orphan blocked in sem_wait. AsynPool is not affected because each of its workers has its own inqueue and no shared read lock.

Minimal reproduction on main without Celery:

import os, time
from billiard.pool import Pool

p = Pool(1, timeout=1)
r = p.apply_async(time.sleep, (30,))
try:
    r.get(timeout=10)
except Exception as e:
    print("hard time limit:", getattr(e, "exc", e))
time.sleep(1.5)  # let the killed worker be replaced
print(p.apply_async(os.getpid).get(timeout=10))  # never returns on main

Bisected: v4.3.0rc1001b9f0 are fine, bd2f803 and later stall. A py-spy dump of the stuck replacement:

    __enter__ (billiard/synchronize.py:115)
    get_payload (billiard/queues.py:394)
    _recv (billiard/pool.py:445)
    receive (billiard/pool.py:473)
    workloop (billiard/pool.py:351)

Tests

  • test_system_exit_from_task_replaces_worker: a task raising SystemExit is reported to the caller and the worker is replaced (fails on main: the same pid keeps serving).
  • test_hard_timeout_does_not_stall_pool: after a hard time limit the replacement worker processes the next job (fails on main with a TimeoutError once the killed worker has been replaced; the test kills the workers itself on failure because terminate() would hang).
  • test_base_exception_propagates from Propagate BaseException raised in pool workers #452 still passes. Full t/unit passes on Linux (fork) and macOS (spawn).

Since bd2f803 (celery#452) the worker loop catches BaseException around the
task call, so the SystemExit raised by the SIGTERM handler during a hard
time limit is reported as a task failure and the worker goes back to
waiting for jobs. Blocking-pool workers wait inside the inqueue's shared
read lock, and the SIGKILL that follows 0.1s later kills the worker while
it holds that lock: the replacement blocks on _rlock forever, the pool
never processes another job, and terminate() blocks on the same lock.

Keep reporting the exception to the caller, but exit the worker
afterwards when it is SystemExit or KeyboardInterrupt, so the pool
replaces it as it did before celery#452.
@auvipy
auvipy self-requested a review September 9, 2026 09:51
@auvipy
auvipy merged commit b1bb092 into celery:main Sep 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants