Skip to content

Allow worker_pool_start_method='spawn' with the asynchronous prefork pool - #10584

Merged
auvipy merged 4 commits into
celery:mainfrom
GangEunzzang:feat/spawn-with-asynpool
Sep 9, 2026
Merged

Allow worker_pool_start_method='spawn' with the asynchronous prefork pool#10584
auvipy merged 4 commits into
celery:mainfrom
GangEunzzang:feat/spawn-with-asynpool

Conversation

@GangEunzzang

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #10339, as discussed there. Related: #6036 (spawn support has been requested there since 2020; with this change it is available for both prefork pool implementations).

#10339 rejects worker_pool_start_method='spawn' whenever the worker uses an event loop, on the assumption that the asynchronous prefork pool cannot work with spawned children. In practice it does: I ran AsynPool with spawned children on Linux and macOS with hard and soft time limits, worker replacement after a hard time limit, --autoscale, --max-tasks-per-child, acks_late + task_reject_on_worker_lost, revoke(terminate=True), pool_restart, grow/shrink, chains/groups/chords, ETA/retry, large payloads and warm/cold shutdown, without a difference to fork (details in #10339 (comment)). The wiring the PR description worried about — the child reporting inqW_fd back to the parent — holds up because spawnv_passfds preserves fd numbers.

The check also had a side effect: it keyed off w.use_eventloop rather than the pool class, so -P solo and -P threads with an async transport were rejected too.

This PR:

  • removes the rejection in Pool.create(), so spawn works with both prefork pool implementations (and no longer blocks solo/threads);
  • updates the CLI help and the docs accordingly, and notes in the setting docs that replacement children have to import the application within worker_proc_alive_timeout;
  • adds smoke tests that run the worker with worker_pool_start_method = "spawn" against the smoke brokers (AsynPool): tasks run in SpawnPoolWorker-* children, and the pool keeps working after a hard time limit kill.

Testing

  • Unit tests updated (test_create_spawn_with_async_pool now asserts the pool is created with forking_enable=False).
  • New smoke tests pass locally.
  • Manual: redis broker with --pool-start-method spawn at -c 2, hard time limit replacement and 50 tasks; -P solo --pool-start-method spawn now starts.

…pool

The check added in celery#10339 rejected 'spawn' whenever the worker uses an
event loop, assuming AsynPool could not work with spawned children. It
does: the fd numbers the children report back to the parent survive
spawnv_passfds, and the pool behaves the same as with fork, including
replacement after a hard time limit. The check was also keyed off
use_eventloop rather than the pool class, so it rejected -P solo and
-P threads with an async transport as well.

Remove it, update the CLI help and docs, and add smoke tests that run
the worker with spawn against the smoke brokers.
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.95%. Comparing base (afeabe2) to head (cb0e646).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10584      +/-   ##
==========================================
+ Coverage   88.91%   88.95%   +0.03%     
==========================================
  Files         153      153              
  Lines       20106    20104       -2     
  Branches     2364     2363       -1     
==========================================
+ Hits        17877    17883       +6     
+ Misses       1923     1919       -4     
+ Partials      306      302       -4     
Flag Coverage Δ
unittests 88.93% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@auvipy
auvipy self-requested a review September 9, 2026 09:35
@auvipy auvipy added this to the 5.7.0 milestone Sep 9, 2026
Comment thread docs/userguide/configuration.rst
@auvipy
auvipy merged commit 7b76598 into celery:main Sep 9, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants