fix(test-optimization): retry payload delivery failures - #10069
Conversation
Overall package sizeSelf size: 8.44 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.4.0 | 127.33 kB | 447.04 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
BenchmarksBenchmark execution time: 2026-09-01 14:36:19 Comparing candidate commit ffeac6d in PR branch Found 1 performance improvements and 0 performance regressions! Performance is the same for 2291 metrics, 18 unstable metrics.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## juan-fernandez/test-optimization-delivery-lifecycle #10069 +/- ##
======================================================================================
Coverage ? 98.58%
======================================================================================
Files ? 993
Lines ? 149650
Branches ? 13145
======================================================================================
Hits ? 147539
Misses ? 2111
Partials ? 0 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
juan-fernandez
changed the title
81eef63 to
8e41f23
Compare
9de57a6 to
b78bb01
Compare
Cap screenshot retries even during finalization, let ready Test Optimization responses beat creation-time timeouts, keep owned agent-discovery retries alive, stabilize Playwright dynamic-name fixtures, and isolate nested integration runs from incomplete GitHub Actions metadata.
b78bb01 to
ffeac6d
Compare
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
6dc4c61
into
juan-fernandez/test-optimization-delivery-lifecycle
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ffeac6d02f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const initializationController = new AbortController() | ||
| const initializationOptions = { signal: initializationController.signal } | ||
| const initializationOptions = { | ||
| deadline: Date.now() + FINAL_FLUSH_TIMEOUT, |
There was a problem hiding this comment.
When the Agent is unavailable while a test framework blocks the event loop for more than 60 seconds, the queued connection error is processed against this already-expired deadline and the request cannot retry even if the Agent is now ready. The exporter then permanently selects the legacy writer and loses EVP features for the session. CiVisibilityExporter.flush() already updates this options object with a fresh shutdown deadline, so setting it during construction reintroduces a startup-relative cutoff that should remain deferred until finalization.
Useful? React with 👍 / 👎.
| signal: initializationController.signal, | ||
| // Test runners await agent discovery before starting. A detached retry can let Node exit | ||
| // while that promise is still pending because promises alone do not keep the event loop alive. | ||
| keepProcessAlive: true, |
There was a problem hiding this comment.
When the local Agent is down or unreachable, each fast ECONNREFUSED schedules another retry and this option keeps every retry timer referenced; because the new deadline also bypasses the ordinary attempt cap, a short or empty test run is held open for roughly the full 60-second deadline. This also prevents the beforeExit path from reaching the existing empty-buffer fast return in CiVisibilityExporter.flush(), so agent discovery should not own the event loop when there is no delivery work to preserve.
Useful? React with 👍 / 👎.
| const retryUntilDeadline = options.deadline !== undefined && options.retryUntilDeadline !== false | ||
| const reachedAttemptLimit = !retryUntilDeadline && attemptIndex >= getMaxAttempts(attemptOptions) | ||
| const reachedUnknownNetworkAttemptLimit = isUnknownNetworkError && attemptIndex >= 2 |
There was a problem hiding this comment.
When a final-flush request receives 429 with a valid zero reset delay—such as Retry-After: 0 or an already-elapsed reset timestamp—this deadline mode removes the attempt cap while the rate-limit branch schedules every retry with setTimeout(..., 0). A persistently rate-limiting intake therefore causes a tight request loop for the remainder of the 60-second finalization window, adding load precisely while the endpoint is throttling; retain a cap or apply a positive backoff for zero-delay 429 responses.
Useful? React with 👍 / 👎.
1 participant
What does this PR do?
Improves retries for two Test Optimization payload delivery failures observed in Web UI end-to-end runs:
Coded permanent failures such as
ENOTFOUNDremain non-retriable, and background requests retain their ordinary attempt cap.Motivation
Web UI end-to-end runs reported
endpoint_payload.requests_errorswith bothstatus_code:N/A,error_type:networkanderror_type:err_dd_test_optimization_request_timeout.The existing retry classifier only retries an allowlist of coded transient network errors, while the generic network telemetry indicates the failure had no usable error code and therefore was not retried. Separately, removing the total attempt timer allowed a request waiting for a socket to remain pending until the final-flush deadline aborted it, without giving retry handling an opportunity to run.
Two full Web UI runs using the isolated retry experiment completed without payload request errors. A Web UI run using the stacked lifecycle implementation still reported request timeouts, motivating restoration of the total attempt timeout and deadline-bounded final retries in this draft.
Additional Notes
This is a draft stacked directly on #10044, which now contains both the delivery lifecycle and error telemetry changes formerly split across #10044 and #10045.
Verification:
git diff --check