Skip to content

APMRP-360 stabilize code origin default test - #7553

Open
P403n1x87 wants to merge 2 commits into
mainfrom
dd/apmrp-360-fix-flaky-code-origin-default-20260820
Open

APMRP-360 stabilize code origin default test#7553
P403n1x87 wants to merge 2 commits into
mainfrom
dd/apmrp-360-fix-flaky-code-origin-default-20260820

Conversation

@P403n1x87

Copy link
Copy Markdown
Contributor

Motivation

Test_Debugger_InProduct_Enablement_Code_Origin_Default_On.test_code_origin_enabled_by_default (tracked as APMRP-360) is flaky, failing intermittently with AssertionError: Expected code origin enabled by default across all Python tracer versions and flask weblog variants.

The setup had a race condition: the tracer's code origin product is started asynchronously and only instruments the framework view functions once enabled. The single request was sent immediately after startup and could be served before the _dd.code_origin.type: entry metadata was attached. Compounding this, the 5s wait was too tight for the full request -> metadata -> flush -> agent -> trace-file chain under CI load.

Changes

  • In setup_code_origin_enabled_by_default, send a warmup request and wait (10s) for a code origin span so instrumentation is fully initialized before the real check.
  • Perform the real check with a generous 30s timeout to absorb slow trace flush/delivery under CI, instead of the global TIMEOUT = 5.
  • Reuse the existing wait_for_code_origin_span helper (which handles threshold computation and state reset) for both the warmup and the assertion, replacing the inlined interfaces.agent.wait_for call.
  • Drop the now-unused interfaces import.

Timeouts are scoped as class constants (_WARMUP_TIMEOUT, _CODE_ORIGIN_TIMEOUT) so other tests still using the global TIMEOUT = 5 are unaffected.

Testing

  • ruff format and ruff check pass on the modified file.
  • mypy --config pyproject.toml tests/debugger/ reports "Success: no issues found in 16 source files".
  • The functional fix is validated end-to-end by the DEBUGGER_INPRODUCT_ENABLEMENT scenario in CI, which exercises this test across the affected Python tracer versions and flask weblog variants.

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

PR by Bits - View session in Datadog

Comment @DataDog to request changes

Co-authored-by: P403n1x87 <20231758+P403n1x87@users.noreply.github.com>
@datadog-datadog-prod-us1

Copy link
Copy Markdown

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

@datadog-datadog-us1-prod

Copy link
Copy Markdown

I can only run on private repositories.

@github-actions

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/debugger/test_debugger_inproduct_enablement.py                    @DataDog/debugger @DataDog/system-tests-core

@P403n1x87
P403n1x87 marked this pull request as ready for review August 24, 2026 14:36
@P403n1x87
P403n1x87 requested review from a team as code owners August 24, 2026 14:36

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1aa5dd0b1d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +237 to +238
self.send_weblog_request("/")
self.code_origin_enabled_by_default = self.wait_for_code_origin_span(timeout=self._CODE_ORIGIN_TIMEOUT)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Capture the trace threshold before sending the checked request

wait_for_code_origin_span computes its threshold from data already received and accepts only later trace files. Because it is invoked after the second synchronous weblog request here, a trace that reaches the agent before this call begins is discarded as pre-existing data, and no further request is made to generate another trace; the test then waits 30 seconds and fails. This regresses the previous ordering, which captured the threshold before issuing the request. Start the wait/threshold before the request, or let the helper accept a request boundary, so the checked request's trace cannot be filtered out.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DataDog fix this

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only run on private repositories.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The new helper records its trace threshold after each request. A fast trace can enter that threshold and cause a false failure after a 30-second wait.

Open Bits AI session

🤖 Datadog Autotest · Commit 1aa5dd0 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

self.wait_for_code_origin_span(timeout=self._WARMUP_TIMEOUT)

self.send_weblog_request("/")
self.code_origin_enabled_by_default = self.wait_for_code_origin_span(timeout=self._CODE_ORIGIN_TIMEOUT)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Capture the trace threshold before the request

The test remains flaky and can wait 30 seconds before it reports a false failure.

Assertion details
  • Input: The agent writes the trace for the second GET request before send_weblog_request() returns.
  • Expected: The check must record the trace threshold before it sends each request.
  • Actual: The helper records the latest trace file after the request returns. It then rejects that file because its number is not above the threshold.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants