Bound full GC passes in tests to one test's allocations - #1847
Merged
Merged
Conversation
tests/contrib/langgraph/test_replay.py tripped the workflow deadlock detector on the 3.10 macOS runner while the first activation was in StateGraph.compile(); the interrupt landed in a weakref callback inside ast.parse, i.e. during a cyclic GC pass. That activation does 3-8 ms of work and imports nothing into the sandbox. The test process heap grows to ~1.7M objects and a gen2 pass over it (160-290 ms locally, several times that on a 3-core runner shared by three xdist workers) runs on whichever thread allocates, so first activations occasionally absorb it inside the 2 s budget. Freezing each test's survivors at teardown keeps later passes to one test's allocations: on tests/contrib (3.10, -n 3) the largest GC pause inside an activation dropped from 291 ms to 20 ms and wall time from 48 s to 33 s. The replay test also asserts nothing is imported into the sandbox after initial workflow load, matching the OpenAI Agents replay test.
Freezing after every test made each test's surviving objects permanent: running tests/worker/test_workflow.py serially, RSS grew from 77 MB to 3.6 GB over 229 tests versus a 490 MB plateau without the hook, and on CI the 2-core ubuntu-arm runners died mid-run with "runner has received a shutdown signal" on every attempt. A single freeze after collection keeps the import-time heap out of every later full pass without accumulating anything: RSS plateaus at 390 MB and a full gc.collect() per test drops from 70 ms mean / 179 ms max to 37 ms / 122 ms on the same subset.
Contributor
Author
|
Reworked in 764aa2b (see updated description): a single post-collection freeze replaces the per-test freeze, which leaked each test's survivors and exhausted the 2-core ubuntu-arm runners. |
DABH
marked this pull request as ready for review
October 2, 2026 02:14
tconley1428
approved these changes
Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
tests/conftest.py: after collection,gc.collect()once and thengc.freeze(), so full GC passes in a test process only scan what tests allocate rather than the ~1.7M objects imported at startup.tests/contrib/langgraph/test_replay.py: assert the replay imports nothing into the sandbox after the initial workflow load, so the first activation stays the few milliseconds of work it is today.Why
test_replaytrips the 2 s deadlock detector during the first activation'sStateGraph.compile(), with the interrupt landing in a GC pass insideast.parse. The activation itself is 3-8 ms of work. A full GC pass over the test process heap is 160-290 ms locally and several times that on a shared 3- or 4-core runner, and it runs on whichever thread happens to allocate, so first activations occasionally absorb one inside the 2 s budget. Freezing the import-time heap halves the pass (70 → 37 ms mean, 179 → 122 ms max over a 51-test subset) without changing anything tests can observe.Freezing once matters. An earlier revision froze after every test, which made each test's survivors permanent (RSS 77 MB → 3.6 GB over 229 serial tests) and killed the ubuntu-arm runners; with a single freeze RSS plateaus at 390 MB, versus 490 MB without the hook.
This is a mitigation, not a cure: a slow enough runner can still trip the detector. The same trip inside
StateGraph.compile()hit three Windows lanes across two PRs on 2026-10-01.