Skip to content

Commit d445359

Browse files
authored
test: Deflake adaptive statistics and browser plugin tests on slow CI runners (#2203)
Two unit tests flake on the Windows CI shard under parallel load ([example run](https://github.com/apify/crawlee-python/actions/runs/33367270395/job/99410412450)): - `test_adaptive_crawling_statistics` asserts each adaptive counter equals exactly 1, but when a slow Chromium launch pushes the browser sub-crawl past the default 60s `request_handler_timeout`, `BasicCrawler` retries the request - correct behavior - and every counter increments again (`assert 2 == 1`). The test now passes a 5-minute `request_handler_timeout`, removing the retry trigger while keeping the exact-count assertions, so a real double-counting regression still fails it. - `test_new_browser` ran `page.goto` with Playwright's default 30s timeout, which the first navigation on a saturated runner can exceed. Raised to 60s, matching `test_browser_pool.py`. Verified with deterministic fault injection: a 70s stall in `BrowserPool.new_page` reproduced the exact CI failure before the fix and 0/3 failures after; a 40s-slow server reproduced the `goto` timeout with the default and passed with 60s. Plus 0/30 failures re-running both modules under `pytest -n auto`. No production code is touched. *✍️ Drafted by Claude Code*
1 parent cc011e2 commit d445359

2 files changed

Lines changed: 4 additions & 1 deletion

File tree

‎tests/unit/browsers/test_playwright_browser_plugin.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ async def test_new_browser(plugin: PlaywrightBrowserPlugin, server_url: URL) ->
4040
assert browser_controller.is_browser_connected
4141

4242
page = await browser_controller.new_page()
43-
await page.goto(str(server_url))
43+
await page.goto(str(server_url), timeout=60_000)
4444

4545
await page.close()
4646
await browser_controller.close()

‎tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -492,6 +492,9 @@ async def test_adaptive_crawling_statistics(test_urls: list[str]) -> None:
492492
crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
493493
rendering_type_predictor=static_only_predictor_no_detection,
494494
result_checker=lambda result: False, # noqa: ARG005 # Intentionally unused argument.
495+
# Generous ceiling for each sub crawler run: a slow browser launch that exceeded it would make `BasicCrawler`
496+
# retry the request and increment every counter below once more.
497+
request_handler_timeout=timedelta(minutes=5),
495498
)
496499

497500
@crawler.router.default_handler

0 commit comments

Comments
 (0)