prefork is not safe anymore - #10339
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a configurable prefork child-process start method to mitigate fork-unsafety in modern dependency stacks (notably gRPC-based transports like GCP Pub/Sub), by allowing prefork workers to start via a fresh-interpreter “spawn” model instead of plain fork().
Changes:
- Add
worker_pool_start_method(fork/spawn, defaultfork) and expose it viacelery worker --pool-start-method. - Wire the setting through worker initialization into the prefork pool so the spawn path uses
billiard.set_start_method('spawn', force=True)and marks children as “fresh interpreters”. - Add validation/docs/tests, including rejecting unsupported combinations (spawn + async-prefork/AsynPool).
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
celery/app/defaults.py |
Adds the worker_pool_start_method default (fork) to worker settings. |
celery/bin/worker.py |
Adds CLI flag --pool-start-method (fork/spawn). |
celery/concurrency/prefork.py |
Implements spawn behavior in prefork pool startup using set_start_method('spawn', force=True) and sets FORKED_BY_MULTIPROCESSING. |
celery/worker/components.py |
Validates/threads the start-method choice into pool instantiation; rejects spawn with async-prefork. |
celery/worker/worker.py |
Plumbs pool_start_method into worker defaults with validation. |
docs/getting-started/backends-and-brokers/gcpubsub.rst |
Recommends spawn for gRPC-based Pub/Sub transport. |
docs/userguide/configuration.rst |
Documents the new worker_pool_start_method setting and constraints. |
docs/userguide/optimizing.rst |
Adds guidance on when to use spawn and its trade-offs. |
t/unit/app/test_defaults.py |
Tests the new default config value and namespace mapping. |
t/unit/concurrency/test_prefork.py |
Tests prefork startup behavior for both fork and spawn cases. |
t/unit/worker/test_components.py |
Tests pool instantiation and rejection behavior for start-method combinations. |
t/unit/worker/test_worker.py |
Tests worker default, explicit spawn, and invalid start method handling. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10339 +/- ##
=======================================
Coverage 88.31% 88.31%
=======================================
Files 153 153
Lines 19740 19749 +9
Branches 2286 2289 +3
=======================================
+ Hits 17433 17442 +9
Misses 2008 2008
Partials 299 299
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
|
is there any related issue for this PR? |
|
Compared to the other proposal, this one is probably the least ambitious one (which I think is something positive in this context). The next steps (ie. future PR) could be:
I am not sure creating another pool is the solution. I think |
|
yeah this seems better |
|
@auvipy -> I will address your feedback in the coming 7 days :) |
This seems like a sensible simplification - it seems to be adding complexity otherwise, since it comes with its own commandline option. |
|
@stuaxo I agree with your proposition. Based on @auvipy comment, celery/billiard#443 is also linked. I was not aware of celery/billiard#443 but seing it, I would say that we should take the macOS case as a reason to migrate all the OS from So, if billiard maintains agree, I would be in favor of closing this PR and replace, in billiard, this: if sys.platform == 'darwin':
# bpo-33725: running arbitrary code after fork() is no longer
# reliable on macOS since macOS 10.14 (Mojave). Use spawn by
# default instead.
# See https://github.com/celery/celery/issues/9894
_default_context = DefaultContext(_concrete_contexts['spawn'])
else:
_default_context = DefaultContext(_concrete_contexts['fork'])With (configure # bpo-33725: running arbitrary code after fork() is no longer
# reliable on macOS since macOS 10.14 (Mojave). Use spawn by
# default instead.
# See https://github.com/celery/celery/issues/9894
#
# bpo-40379: CPython changed the default multiprocessing start method on POSIX away from fork in Python 3.14.
# The issue is titled "multiprocessing's default start method of fork()-without-exec() is broken."
# See https://bugs.python.org/issue40379
_default_context = DefaultContext(_concrete_contexts['spawn'])What do you think? |
auvipy
left a comment
auvipy
left a comment
There was a problem hiding this comment.
should we consider smoke tests for this new feature?
|
with the billiard change, don't we need the changes in celery? or both should be changed |
|
@GangEunzzang can you please thoroughly review this PR and its related billiard issues? it would be of great help |
|
@auvipy I spent some time digging into this and debugging the start-method behavior to see how it works in practice. I tested it on macOS 15 and Linux ( A few things came out of that: 1. The overall approach works Exposing the start method and sending spawned children through the existing execv initialization path ( Blocking pool + Unit tests also pass when merged with current 2. AsynPool + spawn also works I don't think the I tested this both by disabling the check on this branch and by injecting I tried hard/soft time limits, worker replacement, autoscaling, I dug into the fd handling because the concern in the PR description does make sense. The child reports With So restricting the option to 3. I could reproduce the currently exits with: The check is based on If the AsynPool restriction is removed, this goes away with it. Otherwise, I think the check should at least be scoped to 4. macOS + billiard 4.3 exposed a separate issue While debugging this I found another case that initially looked related to this PR, but it reproduces without it. billiard 4.3.0rc1 (celery/billiard#443) defaults to The problem is that Celery doesn't know that happened. Since the explicit The result is that every task fails in I reproduced the same thing on unmodified current It also explains the billiard-only approach mentioned earlier: if billiard simply defaults to I think that detection should use the effective start method ( 5. I also noticed that the bootstep order is Beat → Timer → Hub → Pool → …, so With It works functionally, but it doesn't quite match the "fresh interpreter" behavior described for 6. Unrelated billiard While testing against billiard I traced it back to bd2f803 / celery/billiard#452 changing the worker's A hard time limit first sends SIGTERM, whose handler raises I confirmed the replacement was stuck there with py-spy. It reproduces with fork on Linux; 4.3.0rc1 is fine, current billiard I'll file this separately in billiard as well, with a fix that preserves the #427 behavior but lets the worker exit after reporting 7. A few spawn-specific things I ran into These probably don't need to block the PR, but I think they're worth documenting:
Overall, the core Happy to help with those separately, and I can also put together the AsynPool spawn smoke test if that would be useful. A few questions before I open the follow-ups
|
|
thanks a lot @GangEunzzang I think we can move forward with this PR, and welcome your upcoming PR in both billiard and celery. @superboum you can resume the effort... only one suggestion needed to be committed which I can not push |
|
@GangEunzzang or you can fork this branch and start a new pr for smoke tests and additional changes, if that is easier for you. keeping the existing commits.... so we can consider it for v5.7 |
|
@auvipy Sure, happy to help. I'll start with the #452 regression since that's independent, then work on the spawn-detection issue on the Celery/billiard side. @superboum Would you be okay with me picking this up and continuing the work on this PR? I can add the AsynPool changes and smoke tests on top of the existing work. |
|
It is better I merge this PR and you continue in a separate PR with the needed remaining works |
|
You were faster than me. Thanks a lot for carrying the remaining work. |
thanks for the initial minimal change PR! would you mind referencing the existing open issues related to this? |
…pool (#10584) * Allow worker_pool_start_method='spawn' with the asynchronous prefork pool The check added in #10339 rejected 'spawn' whenever the worker uses an event loop, assuming AsynPool could not work with spawned children. It does: the fd numbers the children report back to the parent survive spawnv_passfds, and the pool behaves the same as with fork, including replacement after a hard time limit. The check was also keyed off use_eventloop rather than the pool class, so it rejected -P solo and -P threads with an async transport as well. Remove it, update the CLI help and docs, and add smoke tests that run the worker with spawn against the smoke brokers. * Use a per-call time limit in the spawn smoke test * Add versionadded to the pool start method section of the optimizing guide --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <auvipy@gmail.com>
preforkis not safe anymoreTable of content:
preforkis unsafepreforkis no longer safepreforkcan be safe againA proof/reproducer showing that
preforkis unsafeSetup a test env:
Some info about the environment:
Write
tasks.py:Write
main.pyStart worker following official grpc recommendations:
Send tasks to worker:
Note that the worker becomes stuck with this log (no more progress is made, worker can't process tasks anymore):
Full trace
Other C lib errors
When sending a SIGUSR2, we get this stacktrace (worker is stuck in grpc code):
It might be tempting to not set the recommended variables, but from our experience processing 60 celery tasks/second, it will take less than a day to hit a similar issue where the worker will hang again. It's just harder to write a reproducer.
Why
preforkis no longer safefork()copies memory, but not the resources that memory points to.When the prefork pool spawns a child, the child receives a copy-on-write snapshot of the parent's address space — every Python object, every module global, every data structure. What it does not receive is the operating-system state those objects refer to.
POSIX is explicit that after fork() in a multithreaded process, only the calling thread survives in the child, and the child may safely call almost nothing until it execs.
Consider a trivial example: the parent opens a file/socket and stores its descriptor in a global list. The child inherits the list, and the integer 7 is still sitting there — but in the child, descriptor 7 either does not refer to the same thing, or refers to a connection now shared with (and concurrently used by) the parent. Read from it and you get an error, a corrupted stream, or a hang. The Python object survived the fork; the resource behind it did not.
Stale resource leakage is not theoretical, it is the default for modern dependency stacks, starting with gRPC.
gRPC runs background C threads and explicitly does not support
fork(). The crucial point for Celery users: you do not have to use gRPC in your tasks to be affected — gRPC is pulled in by infrastructure you almost certainly run. The gcpubsub broker (via google-cloud-pubsub) is built on gRPC, and the OpenTelemetry OTLP exporter uses gRPC.The list of widely-used libraries that are unsafe across fork() — because they either hold native state or run background threads — is long and growing: psycopg/psycopg2 (libpq), SQLAlchemy connection pools, pymongo (which now even warns and calls os.register_at_fork), redis/hiredis, OpenSSL (via cryptography/ssl), confluent-kafka (librdkafka), the numeric stack (numpy/scipy/scikit-learn on OpenBLAS/MKL/OpenMP), CUDA-backed torch/tensorflow, and pure-Python-but-threaded SDKs like the OpenTelemetry SDK, Sentry, and Datadog tracers.
The probability that a real-world Celery worker imports none of these is essentially zero.
Asyncio or gevent are not the way out.
The reason the prefork pool exists is not just parallelism; it is the only model in Python that can safely interrupt running code. You cannot safely inject an exception into a thread (the interpreter only delivers it at a bytecode boundary, never inside a blocking C call), and asyncio/gevent cancellation is strictly cooperative: it works only if the task yields, which blocking or CPU-bound or C-level code never does.
The only mechanism that reliably reclaims a task stuck in a recv(), a C extension, or an infinite loop is killing the OS process running it — which is exactly what prefork's hard time_limit does. Interrupting code is a hard requirement for any real timeout guarantee, and process-kill is the only thing that delivers it.
So "just use a different pool" is not an answer for anyone who needs timeouts.
The existing mitigation hooks cannot close the gap.
The usual advice is "(re)create your unsafe resources in worker_process_init." But this only works for resources you can fully reset, and for some libraries (gRPC chief among them) there is no supported way to reset the inherited native state from inside the forked child.
The second piece of advice points the other way: "defer initializing the control process's own dangerous libraries until after it has forked its executors, so the children fork from a still-clean parent". This can protect the initial batch of executors, since the pool is forked early in startup. But it cannot protect replacements, and replacements are the whole point: the hard time_limit kills a stuck executor and forks a fresh one on demand, during steady-state operation. By then the control process has necessarily initialized exactly the libraries you tried to defer — the broker connection (gcpubsub → gRPC) and the telemetry exporter (OpenTelemetry → gRPC) must be live for the worker to receive and observe work at all. So the replacement executor is forked from a parent that is now irreversibly "dirty," and it inherits the corrupted, fork-unsafe state.
We (at consio.ai) have observed exactly this in our production workload: healthy operation right up to the first hard-timeout kill, then a poisoned child is started and hangs forever due to a deadlock in grpc.
The feature you adopt
preforkfor is the feature that breaks it.Why this trade-off was fine before, and why it is not anymore
The fork-without-exec trade-off was a reasonable bet in its era.
Forking is fast and memory-cheap: the child inherits the already-imported application and configuration via copy-on-write, so workers start instantly and share read-only pages. For a single-threaded process linked against a conservative libc/malloc and a handful of simple Python libraries, forking "just worked," and the performance and simplicity wins were real.
That bet no longer pays off, because the platform and the ecosystem moved underneath it.
Modern application stacks are full of background threads (telemetry, tracing, async clients) and native extensions that hold locks and own kernel resources, precisely the conditions under which fork-without-exec is unsafe.
Python maintainers have now reached the same verdict. After a four-year discussion in bpo-40379, CPython changed the default
multiprocessingstart method on POSIX away fromforkin Python 3.14. The issue is titled "multiprocessing's default start method offork()-without-exec()is broken."The reporter, Itamar Turner-Trauring, opened it by noting that:
PyPy's Michał Górny concurred from hard experience:
The thread contains the exact failure we are describing: ravwojdyla reported that:
In other words, the rest of the ecosystem has already concluded that fork-without-exec is the wrong default. Celery's prefork pool, by hardcoding fork, is now swimming against the language itself: keeping a default that delivers diminishing speed gains in exchange for rising, hard-to-diagnose corruption.
preforkcan be safe againThe fix is to give each child a fresh address space instead of a copy of the parent's.
If the child starts from a clean interpreter, there is no inherited gRPC channel, no frozen lock, no stale
descriptor to corrupt; resources are created fresh and correctly in the child. The way to achieve a clean
address space while still using the fork-based worker model is the spawn start method: fork(), then execv() a brand-new Python interpreter in the child. This preserves everything that makes prefork valuable (a separate, killable OS process per task) while shedding the inherited-state hazard.
This capability already exists in billiard, it is simply not exposed. billiard ships full spawn/forkserver contexts (SpawnContext → popen_spawn_posix), and
set_start_method('spawn', force=True)switches the pool over to them. The prefork pool already threads a forking_enable boolean from BasePool through to on_start, and the worker bootstep hardcodesforking_enable=True, so the value was the only thing standing in the way.AsynPool is not compatible with billiard spawn method
Celery's hub-integrated AsynPool (used with asynchronous transports such as amqp and redis) communicates with workers over file descriptors that are set up in the parent and inherited across fork(); a spawned child cannot reproduce that wiring without a substantial rework.
However, the synchronous transports (including gcpubsub) already run on billiard's plain blocking Pool, which is much closer to the standard-library pool and works naturally with a spawn start method.
So the initial, safe, low-risk scope is: expose the knob, and recommend it for the blocking-pool/synchronous-transport path (the gcpubsub broker docs should explicitly recommend it, since gcpubsub is gRPC-based and therefore one of the most fork-hostile setups in the ecosystem).
Longer term, the truly robust design might be a dedicated "IPC" pool and dropping billiard legacy
Rather than fork-then-exec, the control process and the executor processes would be fully independent from birth (no shared address space at all) communicating over an explicit IPC channel such as a Unix socket. This is the architecture chosen by software with a hard reputation for reliability: Postfix and PostgreSQL both run as cooperating, isolated processes that talk over well-defined channels rather than sharing inherited memory, which is precisely what lets them survive the death of any single worker without corruption. Such a pool would give Celery clean isolation and the hard-kill-for-timeout guarantee by design.
How this PR partially fixes the issue
forkorspawn, but it was hardcoded asforkin Celery. We expose that choice through the new worker_pool_start_method setting (fork/spawn), naming the mechanism rather than a boolean so it leaves room for future methods (e.g. forkserver). It defaults to fork to preserve current behavior (non-breaking, no migration), and is wired through both config and a CLI flag (--pool-start-method).on_startnow callsset_start_method('spawn', force=True)for the spawn case instead of the legacyforking_enable(False)execv path. As described above,
forking_enable(False)is a no-op on CPython 3 (it depends on a Cextension billiard never builds there and merely warns), whereas
set_start_method('spawn')is purePython and takes effect. The fork case keeps calling
forking_enable(True)for backward compatibility reasons.incompatible with the asynchronous prefork pool (AsynPool). Rather than letting that surface later as
an obscure hang/crash, the worker validates as early as the pool type is known and refuses to start,
turning an undiagnosable runtime failure into an upfront, actionable error.
itself (default, trade-offs, the AsynPool caveat); add optimization guidance on when to switch
(threads / fork-unsafe C-extensions like gRPC, psycopg, CUDA) and its costs; and call it out on the
gcpubsub page, since its gRPC driver isn't fork-safe and is the concrete case that motivates spawn.
Testing the PR on the reproducer
First patch the
pyproject.tomlas follow:Then run
uv sync. You may restart the worker and confirm the error is still there.Now, we'll fix the error by configuring the
worker_pool_start_methodparameter:Restart the worker, schedule some workloads. Now the worker does not hang anymore.
Note: this work was done as part of my job at Consio.ai.