Skip to content

ci(integration): stabilize Lambda log snapshots - #824

Merged
joeyzhao2018 merged 5 commits into
mainfrom
BridgeAR/2026-09-01-integration-log-resilience
Sep 2, 2026
Merged

ci(integration): stabilize Lambda log snapshots#824
joeyzhao2018 merged 5 commits into
mainfrom
BridgeAR/2026-09-01-integration-log-resilience

Conversation

@BridgeAR

@BridgeAR BridgeAR commented Sep 1, 2026

Copy link
Copy Markdown
Member

CloudWatch returned successful responses before every invocation log was visible. The runner accepted those partial results as snapshot input.

AWS can initialize a sandbox more than ten seconds before its first invocation. This adds proactive-initialization telemetry and changes cold_start, but the broad snapshots treated both scheduler-owned fields as library regressions. Stored zip snapshots also kept a blank line from the former INIT_START substitution.

The runner now polls until every expected completion record is present and uses one AWS/RIE-aware normalizer for current output and snapshots. AWS removes scheduler-owned fields and historical padding. RIE retains cold_start, and focused trace and metric tests pin proactive-initialization behavior.

The runner also resolves its helper directory before changing directories and keeps successful Serverless diagnostics on stderr. Polling starts immediately instead of waiting for the former fixed 20-second delay.

@datadog-datadog-prod-us1-2

This comment has been minimized.

@BridgeAR
BridgeAR force-pushed the BridgeAR/2026-09-01-integration-log-resilience branch 2 times, most recently from 7152e60 to 3d71476 Compare September 1, 2026 23:51
A successful log fetch could still contain fewer completions than were invoked, and AWS proactive initialization changes cold-start fields without a library behavior change. Both cases made the snapshots depend on platform timing.

The AWS snapshots now wait for every completion and ignore only scheduler-owned fields. RIE and focused telemetry tests retain cold-start and proactive-initialization coverage.
1. A relative runner invocation leaves its helper directory relative, so later directory changes make every log normalization call miss the executable.
2. Successful Serverless diagnostics entered the log payload and could produce false snapshot mismatches.
The former `INIT_START` substitution left an empty line in stored zip snapshots, so current output that deletes the whole line failed every zip-handler comparison.
@BridgeAR
BridgeAR force-pushed the BridgeAR/2026-09-01-integration-log-resilience branch from 3d71476 to b5045c6 Compare September 2, 2026 16:12
@BridgeAR
BridgeAR marked this pull request as ready for review September 2, 2026 17:02
@BridgeAR
BridgeAR requested review from a team as code owners September 2, 2026 17:02

@joeyzhao2018 joeyzhao2018 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. good catch. btw, we are also migrating these real aws-lambda based tests to serverless-e2e-tests

@joeyzhao2018
joeyzhao2018 merged commit a02a46d into main Sep 2, 2026
55 checks passed
@joeyzhao2018
joeyzhao2018 deleted the BridgeAR/2026-09-01-integration-log-resilience branch September 2, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants