[benchmarks] Log the run id and Datadog trace for each sequential-steps run - #3248
Open
shalabhc wants to merge 4 commits into
Open
[benchmarks] Log the run id and Datadog trace for each sequential-steps run#3248shalabhc wants to merge 4 commits into
shalabhc wants to merge 4 commits into
Conversation
Contributor
🦋 Changeset detectedLatest commit: 08567fd The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (605 failed)astro (58 failed):
example (47 failed):
express (49 failed):
fastify (56 failed):
hono (55 failed):
nextjs-turbopack (64 failed):
nextjs-webpack (58 failed):
nitro (56 failed):
nuxt (53 failed):
sveltekit (55 failed):
vite (54 failed):
📦 Local Production (1 failed)nextjs-webpack-stable (1 failed):
📋 Other (108 failed)e2e-vercel-prod-nest (54 failed):
e2e-vercel-prod-tanstack-start (54 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
❌ 📦 Local Production
✅ 🐘 Local Postgres
❌ 📋 Other
✅ vercel-multi-region
|
Contributor
📊 Workflow Benchmarks❌ The benchmark run for Partial results from the failed run: commit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 382767ms → this run 379238ms (Δ -3529ms, -1%) 1020 steps (queue-hop) Cumulative STSO time: main 6857ms → this run 8693ms (Δ +1836ms, +27%) 📜 Previous results (3)75e3abeFri, 31 Jul 2026 18:10:29 GMT · run logs
c784f6cFri, 31 Jul 2026 16:59:19 GMT · run logs
fedfc7aFri, 31 Jul 2026 01:20:43 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
TooTallNate
approved these changes
Jul 31, 2026
TooTallNate
left a comment
TooTallNate
left a comment
Member
There was a problem hiding this comment.
Reviewed the capture, the threading, and the renderer. Approving — this closes a real gap in the histogram section (I've done the deployment-id + time-window hunt this replaces), and the degradation matrix is genuinely complete: every optional field falls back the way the table claims, and the three new renderer tests cover the full/degraded/omitted cases (40/40 pass locally).
One correction worth making before merge — the route comment overclaims what the trace contains. route.ts says the trigger request's span "propagates into the workflow's own spans". That's only true under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default linked mode — where each workflow/step invocation is its own trace root, and the trigger request's trace contains only the workflow.start span plus span links to the execution roots. The link is still useful (one hop via span links, and Datadog renders those), but:
- the comment should say what's actually there, so nobody debugging a weird histogram concludes the run "has no spans" when they open a trigger-only trace; and
- consider making the link a Datadog APM search on
workflow.run.id:<runId>instead of (or alongside) the trace permalink — under linked mode that lands on the run's actual execution spans directly, which is where an STSO investigation actually goes. The run id is already in hand.
Fine to take the comment fix now and the search-link idea as a follow-up — the current link is strictly better than nothing either way.
Verified:
- The capture is correctly optional end-to-end: optional-chained
getActiveSpan()(undefined without OTEL → bare run id), type-guarded on the runner side (typeof data.traceId === 'string'), andsequentialRunsomitted rather than[]when the scenario didn't run — so old artifacts, old deployments, and history re-renders all degrade exactly per the table. - Recording all iterations' run refs matches the data: the histograms aggregate samples across iterations, so with
BENCH_SEQUENTIAL_ITERATIONS > 1the comma-joined list is the honest answer. The doc phrase "both histograms come out of the same iteration" is only true for the default single-iteration config — worth a "(iterations, when configured)" tweak since the code already handles N correctly. - Baseline matching reuses the same methodology/backend/app key the percentile baselines use (
resultKeyFor= the metric key minus the row), so the two sides of the diff can't come from mismatched configurations. - The empty changeset is correct — e2e runner, CI scripts, and a workbench route; nothing published changes.
One robustness nit, non-blocking: runId/traceId are spliced into the markdown link unvalidated. The artifact-trust model is consistent with the rest of this renderer, so it's not a security gap — but a cheap shape check (/^[a-z0-9_]+$/i) before rendering the link would keep a malformed value from a buggy deployment from breaking the whole comment's markdown, which is this section's own failure story.
CI: the 2 failures are the nextjs-webpack canary HMR rebuild-count test and its required-check aggregator — the long-running baseline flake, unreachable from a benchmark-comment change.
shalabhc
changed the title
The route comment claimed the trigger request's span "propagates into the workflow's own spans". That only holds under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default `linked` (packages/core/src/telemetry.ts), where each workflow/step invocation is its own trace root and the trigger's trace carries `workflow.start` plus span links out to those roots. The logged link is still the right entry point -- one hop through the links, which Datadog renders -- but the comment should describe that, so nobody opening a trigger-only trace while debugging a histogram concludes the run produced no spans. Raised by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
Under the default linked trace mode the trigger's trace holds only `workflow.start` plus span links, so opening it lands one hop away from the spans an STSO investigation needs. Log an APM search on `@workflow.run.id:<runId>` next to it, which goes straight to the run's execution spans. Both links are logged rather than one replacing the other: the search depends on `workflow.run.id` being an indexed span tag in the org, and the permalink works regardless. Suggested by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Logs the run id and Datadog APM links for each sequential-steps benchmark iteration, in the benchmark job's own output:
Follow-up to #3213 (which added the STSO histograms). Uses the run-id/trace capture prototyped on the WIP variance branch, #3107.
Why
The
📈 STSO distributionsection shows the shape of the sequential-steps run but not which run produced it. When a bucket looks wrong, the next step is always the same manual hunt: find the run by deployment id + time window, then find its trace. This puts both in the log, next to the run that produced them.These are debugging aids rather than part of the benchmark's reported result, so they go in the job output rather than the PR comment body. That also covers two cases a comment can't: a local
pnpm bench, and a job that fails before the comment step runs.What
workbench/nextjs-turbopack/app/api/bench/route.ts— returnstraceIdfrom the span@vercel/otelopened for the trigger request.@opentelemetry/apiwas already a dependency; no lockfile change.packages/core/e2e/benchmark.test.ts— threadstraceIdthroughtriggerBenchRun→runSequentialIteration, and logs one line per iteration as soon as the scenario completes.A deployment built before the route reported a trace id logs the run id and span search, with no trace permalink.
Why two links
This deployment runs the default
WORKFLOW_TRACE_MODE=linked— nothing in the workbench orbenchmarks.ymlsets the mode. Underlinked, each workflow/step invocation is its own trace root, and the trigger's trace holds onlyworkflow.startplus span links out to those roots. So the trace permalink is an entry point to the run (one hop through the links, which Datadog renders), not a single trace containing every step's spans.The
@workflow.run.id:<runId>span search lands directly on the run's execution spans instead, which is where an STSO investigation actually goes. It depends onworkflow.run.idbeing an indexed span tag in the org, so it's logged alongside the permalink rather than replacing it — still to be confirmed against a real run.Notes
The comment renderer is untouched, and the result artifact gains no new fields — the run identities are logged, not serialized.
One thing this deliberately drops relative to putting the line in the comment: there's no link to the
mainbaseline's run. Which run produced the baseline histogram is only knowable at comment-render time, where the two artifacts are matched up; the benchmark job only knows about its own run.Empty changeset — no package behavior changes.
🤖 Generated with Claude Code