[benchmarks] Variance analysis + follow-up work - #3236
Conversation
Records the run-to-run noise floor of each benchmark metric, measured against five real CI runs rather than inferred: four preview runs of byte-identical runtime code plus the production baseline they were diffed against. The core result is that the measurement is precise but the deployments are not — two runs of identical code agreed to within 0.2% on cumulative inline STSO, while a third was 8% slower across its entire distribution. That sets a practical detection floor (inline STSO ±8%; TTFS Best not usable at all, being a min over 30 samples drawn from a bimodal population), which is why a no-op PR tripped a P99 regression flag on 15 samples out of 1016. Also records the hypotheses this data refuted, so they aren't re-proposed — notably that preview-vs-production explains the inline differences (it does not; production sits inside the preview range) — and five follow-up items, including the environment mismatch that gives every PR a built-in ~+25% TTFS regression. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
🦋 Changeset detectedLatest commit: ea1a046 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 440353ms → this run 408215ms (Δ -32138ms, -7%) 1020 steps (queue-hop) Cumulative STSO time: main 9664ms → this run 8160ms (Δ -1504ms, -16%) 📜 Previous results (1)cc0c16fThu, 30 Jul 2026 22:31:48 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Adds eight more CI runs, including three consecutive production runs so main can be compared against main, and three more runtime-identical preview runs (a no-op line, a docs-only PR, a documentation-only PR). Two findings this data adds: Production is no more stable than preview. Two main runs twelve minutes apart, differing only by a lint-only commit, came out 13% apart (388.5s vs 440.4s); the production null spread (12.6%) matches preview's (13.2%). That kills the last version of the "it's the environment" explanation for inline STSO, leaving the deployment draw itself. The queue-hop tag has a blind spot: it keys off process identity, so a hop onto a reused warm instance is tagged inline. Runs reporting zero hops still carry exactly three multi-second inline samples, and one production run reporting three hops carried two more. Actual boundary counts run 3-7 where the reported count runs 0-3, so the inline population is contaminated on an unpredictable subset of runs. Noted at the tagging site in 97_bench.ts as well. Also revises the detection floor: the earlier ±8% came from the first four runs and understated it. Six runtime-identical preview runs give sd ≈ 5% and an observed range of 13.2%. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
The earlier "3-7 actual boundaries" was an artifact of treating every inline sample over 1.5s as a mislabeled hop. The boundary count is not a matter of inference: an invocation is capped at ~120s, so a run of duration WO must cross ceil(WO/120s)-1 of them, which is exactly 3 for every run in the set except the broken repl-det branch (11). So the runs reporting 3 hops already found all of them, and their extra multi-second inline samples are genuine stalls rather than mislabeled hops. The >1.5s heuristic only finds hops on runs that reported none. Detection turns out to be all-or-nothing — 3 of 3 or 0 of 3, never in between, across eleven runs — which points at a per-run property rather than a per-boundary coin flip. Also records the decisive test now that #3186 persists the compute instance per step attempt: group one 0-hop and one 3-hop run's attempts by instance id. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
1 participant
Follow-up to #3213. Starts with the analysis itself so it is durable and reviewable; the code changes it recommends land on this branch after.
What's here now
packages/core/e2e/BENCHMARK_VARIANCE.md— the measured run-to-run noise floor of each benchmark metric, derived from twelve real CI runs across both environments.benchmark.test.tsand97_bench.tsgain pointers to it.The dataset is the useful part: six preview runs with byte-identical runtime code (renderer-only commits on #3213, a
// NO-OPline on #3101, docs-only on #3170 and #3236), plus three consecutive production runs somaincan be compared againstmain.Two headline results
1. Production is no more stable than preview — it's the deployment draw.
Production null spread 12.6%, preview 13.2%. Ranges overlap and the single slowest of all twelve runs is a production run. Two runs of identical code agreed to 0.2%, so the instrument is precise — the deployments differ.
2. The queue-hop tag has a blind spot, and it contaminates the inline population.
It keys off process identity, so a hop onto a reused warm instance is tagged
inline. Runs reporting zero hops still carry exactly three multi-second inline samples:Actual boundary counts run 3-7 where the reported count runs 0-3.
COMPUTE_INSTANCE_IDhas the same blind spot by construction; the only invocation-scoped id in the tree isrequestIdStorageinworld-vercel/src/queue.ts, which step bodies can't see.Detection floor
For scale, a branch with a genuine regression measured an inline mean of 1361 ms against a null mean of 388 ms. The metric catches real regressions comfortably; it's the sub-15% range that can't be trusted.
Refuted hypotheses (recorded so they don't get re-proposed)
One environment effect is real: TTFS p50 is 966-1063 ms on production but 1240-1384 ms on preview, so every PR carries a built-in ~25% TTFS regression from the
main-runs-production / PRs-run-preview split.Follow-up items (in the doc, to land on this branch)
rawin run order — stored sorted today, which makes the event-log-growth hypothesis for the slow tail untestable. One line, no output change.main, not production.refs/pull/N/mergesha despite the step env holding the head sha).Empty changeset — no package behavior changes.
🤖 Generated with Claude Code