From cc0c16fa415ec638b466ac85a96d1eebd6e2b55d Mon Sep 17 00:00:00 2001
From: "vercel[bot]" <35613825+vercel[bot]@users.noreply.github.com>
Date: Thu, 30 Jul 2026 22:07:23 +0000
Subject: [PATCH 1/3] Document measured benchmark variance and its sources
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Records the run-to-run noise floor of each benchmark metric, measured
against five real CI runs rather than inferred: four preview runs of
byte-identical runtime code plus the production baseline they were
diffed against.

The core result is that the measurement is precise but the deployments
are not — two runs of identical code agreed to within 0.2% on cumulative
inline STSO, while a third was 8% slower across its entire distribution.
That sets a practical detection floor (inline STSO ±8%; TTFS Best not
usable at all, being a min over 30 samples drawn from a bimodal
population), which is why a no-op PR tripped a P99 regression flag on 15
samples out of 1016.

Also records the hypotheses this data refuted, so they aren't
re-proposed — notably that preview-vs-production explains the inline
differences (it does not; production sits inside the preview range) —
and five follow-up items, including the environment mismatch that gives
every PR a built-in ~+25% TTFS regression.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---
 .changeset/benchmark-variance-notes.md  |   4 +
 packages/core/e2e/BENCHMARK_VARIANCE.md | 247 ++++++++++++++++++++++++
 packages/core/e2e/benchmark.test.ts     |   5 +
 3 files changed, 256 insertions(+)
 create mode 100644 .changeset/benchmark-variance-notes.md
 create mode 100644 packages/core/e2e/BENCHMARK_VARIANCE.md

diff --git a/.changeset/benchmark-variance-notes.md b/.changeset/benchmark-variance-notes.md
new file mode 100644
index 0000000000..53b095437a
--- /dev/null
+++ b/.changeset/benchmark-variance-notes.md
@@ -0,0 +1,4 @@
+---
+---
+
+Document the measured run-to-run noise floor of each benchmark metric, and the deployment-level effects behind it, in `packages/core/e2e/BENCHMARK_VARIANCE.md`.
diff --git a/packages/core/e2e/BENCHMARK_VARIANCE.md b/packages/core/e2e/BENCHMARK_VARIANCE.md
new file mode 100644
index 0000000000..cf0eadba55
--- /dev/null
+++ b/packages/core/e2e/BENCHMARK_VARIANCE.md
@@ -0,0 +1,247 @@
+# Benchmark variance analysis
+
+_Analysis date: 2026-07-30. Data: five CI benchmark runs of `nextjs-turbopack` on
+the `vercel` backend, methodology version 2._
+
+This documents where run-to-run variance in the Performance Benchmarks job
+actually comes from, measured against real CI data rather than inferred. It
+exists because the benchmark comment was reporting large deltas — up to +523%
+— on changes that could not possibly have caused them, which made the whole
+report easy to dismiss.
+
+The headline: **the measurement is precise, but each run measures a different
+deployment, and deployments differ from each other by more than most real
+regressions would.** Two runs of identical code agreed to within 0.2%; a third
+was 8% slower across its entire distribution.
+
+## Detection floor (the practical takeaway)
+
+Measured as the spread across four preview runs of **identical runtime code**.
+A delta smaller than this is not evidence of anything.
+
+| Signal | Null spread | Verdict |
+|---|---|---|
+| inline STSO — cumulative / mean | ±8% | usable above ~8% |
+| inline STSO — p50 | 365-409 ms (±6%) | usable, similar |
+| inline STSO — p99 | 731-790 ms (±8%) | the ±15% 🔻 threshold is only ~2× the noise |
+| inline STSO — tail count (>700 ms) | 12-23 samples | very noisy, do not gate on it |
+| queue-hop STSO — count | exactly 3, every run | stable |
+| queue-hop STSO — per-hop cost | 1545-3446 ms (n=3/run) | too few samples to read |
+| TTFS — p50 | ±3% within an environment | usable, but see the preview/production bias below |
+| TTFS — Best | 0-11 fast samples out of 30 | not a usable signal |
+
+## Dataset
+
+All five runs execute the same workload: `benchSequentialStepsWorkflow(1020)`,
+one iteration, yielding 1016 inline STSO samples and 3 queue-hop samples each.
+
+| Run | When (UTC) | Env | CI run | Head SHA |
+|---|---|---|---|---|
+| PR #3213 | 18:30 | preview | [30570697187](https://github.com/vercel/workflow/actions/runs/30570697187) | `95cf46a` |
+| PR #3213 | 18:58 | preview | [30572791924](https://github.com/vercel/workflow/actions/runs/30572791924) | `1bfc037` |
+| PR #3213 | 19:34 | preview | [30575442330](https://github.com/vercel/workflow/actions/runs/30575442330) | `342f5a3` |
+| `main` (post-merge) | 21:04 | **production** | [30580048090](https://github.com/vercel/workflow/actions/runs/30580048090) | `8bda7ce` |
+| PR #3101 (no-op) | 21:30 | preview | [30582183243](https://github.com/vercel/workflow/actions/runs/30582183243) | `32225d9` |
+
+The three PR #3213 runs are the important control: their later commits touched
+only `.github/scripts/render-benchmark-comment.mjs` and its tests, so **the
+runtime code and the benchmark workload were byte-identical across all three**.
+PR #3101 adds a single `// NO-OP` comment line to `packages/core/src/runtime.ts`
+and is likewise runtime-identical.
+
+## The data
+
+### Inline STSO (n = 1016 per run)
+
+```
+run                    cum(s)   mean   p10   p50   p90   p99    max  tail>700  body<=700
+#3213 @18:30 (prev)     386.0    380   238   378   515   743   1232      12       374
+#3213 @18:58 (prev)     386.8    381   235   376   517   731   1179      15       374
+#3213 @19:34 (prev)     416.7    410   257   409   554   736   1068      16       404
+#3101 @21:30 (prev)     392.4    386   241   365   531   790   1328      23       375
+main  @21:04 (PROD)     408.2    402   253   406   535   684   1044       8       398
+```
+
+`tail>700` is the count of samples above 700 ms; `body<=700` is the mean of
+everything at or below it.
+
+### TTFS (step scenario, n = 30) and queue-hops
+
+```
+run                  fast<600ms    min    p50   | hops  values (ms)
+#3213 @18:30 (prev)        0/30   1267   1300   |    3  [2270, 3013, 3446]
+#3213 @18:58 (prev)        0/30   1275   1304   |    3  [2256, 3107, 3125]
+#3213 @19:34 (prev)       11/30    273   1308   |    3  [1545, 2125, 2632]
+#3101 @21:30 (prev)        0/30   1308   1340   |    3  [2160, 3108, 3168]
+main  @21:04 (PROD)        6/30    210   1039   |    3  [1628, 2384, 2995]
+```
+
+`fast<600ms` counts samples in the fast TTFS mode (see finding 3).
+
+## Findings
+
+### 1. The measurement is precise; the deployments are not
+
+The 18:30 and 18:58 runs — separate preview deployments, separate 1020-step
+executions — agree to **0.2%** on cumulative inline time (386.0 s vs 386.8 s)
+and **0.3%** on body mean (374 vs 374). A 1016-sample run is a stable
+instrument.
+
+So the between-run differences are not sampling noise. They are real
+differences between the deployments being measured.
+
+### 2. One deployment was simply slower, across its whole distribution
+
+The 19:34 run sits to the right at every quantile: p10 257 vs ~236, p50 409 vs
+~377, body mean 404 vs ~374. That is +8%, and it is not a tail artifact — the
+entire body moved.
+
+This is the single largest source of inline variance, and nothing about the
+workload explains it. Some deployments land slow and stay slow for the whole
+run. Suspects (unconfirmed): instance placement, neighbour load on shared
+infrastructure, pool warmth at allocation time.
+
+### 3. Body and tail are separate signals that move independently
+
+For PR #3101 vs the `main` baseline, the comment flagged `P99 +15% 🔻` — but:
+
+- body (≤700 ms, ~98% of samples): mean 375 vs 398, i.e. **6% faster**
+- tail (>700 ms): 23 samples vs 8, total 19.7 s vs 6.7 s
+- cumulative: −4%, which is a −29 s body improvement plus a +13 s tail regression
+
+**The 🔻 was driven by 15 samples out of 1016.** Across the preview cohort the
+tail count ranges 12-23 and prod's 8 was the cohort minimum, so the flag was
+"fattest preview tail vs thinnest production tail," not a change in step cost.
+
+Whether the tail is concentrated late in the run (growing event log) or
+scattered is **currently untestable** — see open item 2.
+
+### 4. TTFS is bimodal, and mode occupancy is a lottery
+
+TTFS samples fall into a fast mode (~200-400 ms, the in-process fast path) and a
+slow mode (~1300 ms). How many of a run's 30 samples land in the fast mode is
+close to random: 0, 0, **11**, 0 across previews and 6 on production.
+
+Because **Best is a min over 30 samples**, a single lucky sample moves it 5×.
+That is the mechanism behind the `+523% 🔻` on a no-op change. Best is not a
+usable regression signal in its current form; the fast-mode *fraction* would be.
+
+### 5. Preview vs production is a real bias — but only for TTFS median
+
+```
+TTFS(step) p50:  preview 1300, 1304, 1308, 1340   |   production 1039
+```
+
+The four preview medians agree within 3%; production is **20% faster than all
+of them**. Since `main` runs against production
+(`environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'preview' }}`,
+`.github/workflows/benchmarks.yml`) and every PR runs against preview, **every
+PR carries a built-in ~+25% TTFS "regression" that is pure environment
+mismatch**.
+
+### 6. The queue-hop count is stable
+
+Every run crossed exactly 3 invocation boundaries — the ~120 s per-invocation
+ceiling and the total run duration are consistent. Only the per-hop cost varies
+(1545-3446 ms), and with n=3 per run those percentiles are degenerate: p75, p90
+and p99 all collapse to the max sample.
+
+## Hypotheses this data refuted
+
+Recorded so they are not re-proposed:
+
+- **"Preview vs production explains the inline STSO differences."** It does not.
+  Production (408.2 s) sits *inside* the preview range (386-417 s); one preview
+  run was slower than production. The environment split is a real bias for TTFS
+  p50 only.
+- **"The TTFS fast mode is a production-only effect."** No — the 19:34 preview
+  run hit it 11 times out of 30, more often than production did.
+- **"WO is an independent whole-run signal."** It is not. `WO − Σ(all STSO gaps)`
+  is 1019 ms (main) and 1362 ms (this) — WO is the gap total plus step-body
+  time, so it carries nothing the STSO rows do not already.
+
+## Open items
+
+1. **Baseline against a preview deployment of `main`, not production.** Removes
+   the built-in TTFS bias (finding 5). Requires `main`'s benchmark job to deploy
+   and target its own preview rather than switching on `github.ref`.
+2. **Keep `raw` in run order.** `computeStats` currently stores `raw: sorted`,
+   which destroys ordering at write time, so "is the slow tail concentrated in
+   the last 200 steps?" — the obvious event-log-growth hypothesis — cannot be
+   tested from artifacts. Fix: store insertion order, sort locally inside
+   `computeStats` for the percentiles. No output changes.
+3. **Loosen or replace the p99 threshold for inline STSO.** At ±8% null spread,
+   a ±15% 🔻 will keep crying wolf. Body mean or p50 is the tighter signal.
+4. **Report TTFS fast-mode fraction instead of (or beside) Best.** Best is a min
+   over 30 samples and therefore a lottery; the fraction is the real signal.
+5. **Fix commit provenance on PR runs.** `.github/workflows/benchmarks.yml` sets
+   `GITHUB_SHA` to the PR head sha for the benchmark step, and the job log
+   confirms the step env holds it (`342f5a3a5…`) — but the written artifact
+   records the synthetic `refs/pull/N/merge` sha (`9dfa2b64…`) instead. Root
+   cause not yet identified. Harmless for numbers, wrong for traceability.
+
+## Reproducing this analysis
+
+Download the artifacts for any two runs (must be run from inside a clone):
+
+```bash
+gh run download <RUN_ID> --pattern 'bench-results-*' --dir /tmp/bench/<label>
+```
+
+Each `bench-results-*.json` carries the full sample arrays (`metrics[].raw`), so
+no Datadog access is needed for distribution work. Compare them with:
+
+```js
+// node -e '<this>'
+const fs = require('node:fs'), path = require('node:path');
+const load = (d) => {
+  const out = [];
+  (function walk(x) {
+    for (const e of fs.readdirSync(x, { withFileTypes: true })) {
+      const f = path.join(x, e.name);
+      if (e.isDirectory()) walk(f);
+      else if (/bench-results-.*\.json$/.test(e.name))
+        out.push(JSON.parse(fs.readFileSync(f, 'utf8')));
+    }
+  })(d);
+  return out;
+};
+const row = (r, m, s) => r.metrics.find((x) => x.metric === m && x.scenario === s);
+const q = (a, p) => a[Math.min(a.length - 1, Math.ceil(p * a.length) - 1)];
+const sum = (a) => a.reduce((x, y) => x + y, 0);
+const mean = (a) => sum(a) / a.length;
+
+for (const [name, dir] of [['A', '/tmp/bench/a'], ['B', '/tmp/bench/b']]) {
+  const a = row(load(dir)[0], 'stso', '1020 steps (inline)').raw;
+  console.log(
+    name,
+    'cum', (sum(a) / 1000).toFixed(1) + 's',
+    'mean', mean(a).toFixed(0),
+    'p50', q(a, 0.5),
+    'p99', q(a, 0.99),
+    'tail>700', a.filter((v) => v > 700).length,
+    'body<=700', mean(a.filter((v) => v <= 700)).toFixed(0)
+  );
+}
+```
+
+Note that `raw` is stored **sorted**, so quantile and histogram comparisons work
+but anything order-dependent does not (open item 2).
+
+## Background: what the metrics mean
+
+`STSO` is split by whether the step ending a gap ran **inline** (same warm
+process as the step before it — pure framework overhead) or after a
+**queue-hop** (first step of a fresh process, paying queue dispatch, client
+reinit and event-log replay). The workflow tags each step itself via a
+process-global in `workbench/example/workflows/97_bench.ts`; the split is ground
+truth, not inferred from step index or trace timestamps.
+
+That split was introduced in [#3213](https://github.com/vercel/workflow/pull/3213)
+precisely because the previous step-index windows (`1-20` / `101-120` /
+`1001-1020`, 19 samples each) mixed the two populations: whether an invocation
+boundary happened to land inside a window moved that window's P99 by hundreds of
+percent between identical runs. The analysis above is what became visible once
+that noise source was removed.
+
+See `packages/core/e2e/benchmark.test.ts` for the full metric definitions.
diff --git a/packages/core/e2e/benchmark.test.ts b/packages/core/e2e/benchmark.test.ts
index 312dae14b2..b25269fca2 100644
--- a/packages/core/e2e/benchmark.test.ts
+++ b/packages/core/e2e/benchmark.test.ts
@@ -2,6 +2,11 @@
  * Benchmark runner measuring the workflow runtime's core latency metrics
  * against a deployed workbench app.
  *
+ * Before reading a delta in the PR comment as a regression, see
+ * BENCHMARK_VARIANCE.md in this directory: it measures the run-to-run noise
+ * floor of each metric against real CI data (inline STSO ±8%, TTFS Best not
+ * usable at all), and records which apparent signals are artifacts.
+ *
  * Every run is triggered through an in-deployment route (`POST /api/bench` on
  * the workbench app) rather than by calling `start()` from this CI process. The
  * route stamps `clientStart` with the deployment's own clock immediately before

From 0207a58c9557202100f8d72b82320073f1acb14a Mon Sep 17 00:00:00 2001
From: "vercel[bot]" <35613825+vercel[bot]@users.noreply.github.com>
Date: Thu, 30 Jul 2026 23:17:21 +0000
Subject: [PATCH 2/3] Update variance analysis with production and warm-reuse
 findings
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Adds eight more CI runs, including three consecutive production runs so
main can be compared against main, and three more runtime-identical
preview runs (a no-op line, a docs-only PR, a documentation-only PR).

Two findings this data adds:

Production is no more stable than preview. Two main runs twelve minutes
apart, differing only by a lint-only commit, came out 13% apart (388.5s
vs 440.4s); the production null spread (12.6%) matches preview's
(13.2%). That kills the last version of the "it's the environment"
explanation for inline STSO, leaving the deployment draw itself.

The queue-hop tag has a blind spot: it keys off process identity, so a
hop onto a reused warm instance is tagged inline. Runs reporting zero
hops still carry exactly three multi-second inline samples, and one
production run reporting three hops carried two more. Actual boundary
counts run 3-7 where the reported count runs 0-3, so the inline
population is contaminated on an unpredictable subset of runs. Noted at
the tagging site in 97_bench.ts as well.

Also revises the detection floor: the earlier ±8% came from the first
four runs and understated it. Six runtime-identical preview runs give
sd ≈ 5% and an observed range of 13.2%.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---
 packages/core/e2e/BENCHMARK_VARIANCE.md | 348 +++++++++++++++---------
 packages/core/e2e/benchmark.test.ts     |   5 +-
 workbench/example/workflows/97_bench.ts |   9 +
 3 files changed, 238 insertions(+), 124 deletions(-)

diff --git a/packages/core/e2e/BENCHMARK_VARIANCE.md b/packages/core/e2e/BENCHMARK_VARIANCE.md
index cf0eadba55..7d8189ac7b 100644
--- a/packages/core/e2e/BENCHMARK_VARIANCE.md
+++ b/packages/core/e2e/BENCHMARK_VARIANCE.md
@@ -1,7 +1,8 @@
 # Benchmark variance analysis
 
-_Analysis date: 2026-07-30. Data: five CI benchmark runs of `nextjs-turbopack` on
-the `vercel` backend, methodology version 2._
+_Analysis date: 2026-07-30. Data: twelve CI benchmark runs of `nextjs-turbopack`
+on the `vercel` backend, methodology version 2, spanning both the preview
+(PR) and production (`main`) environments._
 
 This documents where run-to-run variance in the Performance Benchmarks job
 actually comes from, measured against real CI data rather than inferred. It
@@ -9,99 +10,175 @@ exists because the benchmark comment was reporting large deltas — up to +523%
 — on changes that could not possibly have caused them, which made the whole
 report easy to dismiss.
 
-The headline: **the measurement is precise, but each run measures a different
-deployment, and deployments differ from each other by more than most real
-regressions would.** Two runs of identical code agreed to within 0.2%; a third
-was 8% slower across its entire distribution.
+Two headline results:
+
+1. **The measurement is precise, but each run measures a different deployment**,
+   and deployments differ from each other by more than most real regressions
+   would. This holds in production as much as in preview: two `main` runs twelve
+   minutes apart, differing only by a lint-only commit, came out 13% apart.
+2. **The inline/queue-hop split has a blind spot.** It tags by *process*
+   identity, so a queue-hop onto a *reused warm* process is silently counted as
+   an inline step. Affected runs carry multi-second samples inside the inline
+   population.
 
 ## Detection floor (the practical takeaway)
 
-Measured as the spread across four preview runs of **identical runtime code**.
-A delta smaller than this is not evidence of anything.
+Measured across six preview runs of **byte-identical runtime code** and three
+consecutive production runs. A delta smaller than this is not evidence of
+anything.
 
 | Signal | Null spread | Verdict |
 |---|---|---|
-| inline STSO — cumulative / mean | ±8% | usable above ~8% |
-| inline STSO — p50 | 365-409 ms (±6%) | usable, similar |
-| inline STSO — p99 | 731-790 ms (±8%) | the ±15% 🔻 threshold is only ~2× the noise |
-| inline STSO — tail count (>700 ms) | 12-23 samples | very noisy, do not gate on it |
-| queue-hop STSO — count | exactly 3, every run | stable |
-| queue-hop STSO — per-hop cost | 1545-3446 ms (n=3/run) | too few samples to read |
-| TTFS — p50 | ±3% within an environment | usable, but see the preview/production bias below |
-| TTFS — Best | 0-11 fast samples out of 30 | not a usable signal |
+| inline STSO — cumulative / mean | sd ≈ 5% of mean; **observed range 13%** | treat ±10% as the floor |
+| inline STSO — p99 | 661-835 ms (±12%) | the ±15% 🔻 threshold is barely above noise |
+| inline STSO — tail count (>700 ms) | 8-28 samples | do not gate on it |
+| queue-hop STSO — count | **0-3 reported, 3-7 actual** | see finding 3; currently unreliable |
+| queue-hop STSO — per-hop cost | 1545-4663 ms (n≤3/run) | too few samples to read |
+| TTFS — p50 | ±3% within an environment | usable; see the environment bias in finding 5 |
+| TTFS — Best | 0-17 fast samples out of 30 | not a usable signal |
+
+For scale: a branch with a genuine regression (`pgp/replay-engine-determinism-tests`,
+22:15) measured an inline mean of **1361 ms against a null mean of 388 ms** —
+3.5×. The metric detects real regressions comfortably; it is the sub-15% range
+where it cannot be trusted.
 
 ## Dataset
 
-All five runs execute the same workload: `benchSequentialStepsWorkflow(1020)`,
-one iteration, yielding 1016 inline STSO samples and 3 queue-hop samples each.
+All runs execute the same workload: `benchSequentialStepsWorkflow(1020)`, one
+iteration, yielding 1019 STSO samples split between the two populations.
 
-| Run | When (UTC) | Env | CI run | Head SHA |
-|---|---|---|---|---|
-| PR #3213 | 18:30 | preview | [30570697187](https://github.com/vercel/workflow/actions/runs/30570697187) | `95cf46a` |
-| PR #3213 | 18:58 | preview | [30572791924](https://github.com/vercel/workflow/actions/runs/30572791924) | `1bfc037` |
-| PR #3213 | 19:34 | preview | [30575442330](https://github.com/vercel/workflow/actions/runs/30575442330) | `342f5a3` |
-| `main` (post-merge) | 21:04 | **production** | [30580048090](https://github.com/vercel/workflow/actions/runs/30580048090) | `8bda7ce` |
-| PR #3101 (no-op) | 21:30 | preview | [30582183243](https://github.com/vercel/workflow/actions/runs/30582183243) | `32225d9` |
+**Production (`main`), consecutive merges:**
 
-The three PR #3213 runs are the important control: their later commits touched
-only `.github/scripts/render-benchmark-comment.mjs` and its tests, so **the
-runtime code and the benchmark workload were byte-identical across all three**.
-PR #3101 adds a single `// NO-OP` comment line to `packages/core/src/runtime.ts`
-and is likewise runtime-identical.
+| When (UTC) | Commit | What landed | CI run |
+|---|---|---|---|
+| 20:38 | `8bda7ce` | #3213 — the inline/queue-hop split itself | [30580048090](https://github.com/vercel/workflow/actions/runs/30580048090) |
+| 22:20 | `4a9d26b` | #3186 — persist compute instance per step attempt | [30586783627](https://github.com/vercel/workflow/actions/runs/30586783627) |
+| 22:32 | `32ac8e7` | #3222 — Biome lint fixes (semantically neutral) | [30587485157](https://github.com/vercel/workflow/actions/runs/30587485157) |
 
-## The data
+**Preview (PRs) with runtime code byte-identical to `8bda7ce`** — the null set:
 
-### Inline STSO (n = 1016 per run)
+| When (UTC) | PR | Why it is runtime-identical | CI run |
+|---|---|---|---|
+| 18:30 | #3213 | later commits touched only the comment renderer | [30570697187](https://github.com/vercel/workflow/actions/runs/30570697187) |
+| 18:58 | #3213 | " | [30572791924](https://github.com/vercel/workflow/actions/runs/30572791924) |
+| 19:34 | #3213 | " | [30575442330](https://github.com/vercel/workflow/actions/runs/30575442330) |
+| 21:30 | #3101 | adds one `// NO-OP` comment line | [30582183243](https://github.com/vercel/workflow/actions/runs/30582183243) |
+| 22:07 | #3236 | documentation only | [30586031837](https://github.com/vercel/workflow/actions/runs/30586031837) |
+| 22:30 | #3170 | `docs/` app only (35 files, no runtime) | [30587369865](https://github.com/vercel/workflow/actions/runs/30587369865) |
+
+**Preview runs with real code changes** (context only, not part of the null set):
+#3238 `pgp/bisect-3198-only` ([30588202433](https://github.com/vercel/workflow/actions/runs/30588202433)),
+`pgp/replay-engine-determinism-tests` ([30586473644](https://github.com/vercel/workflow/actions/runs/30586473644)).
+
+## The data
 
 ```
-run                    cum(s)   mean   p10   p50   p90   p99    max  tail>700  body<=700
-#3213 @18:30 (prev)     386.0    380   238   378   515   743   1232      12       374
-#3213 @18:58 (prev)     386.8    381   235   376   517   731   1179      15       374
-#3213 @19:34 (prev)     416.7    410   257   409   554   736   1068      16       404
-#3101 @21:30 (prev)     392.4    386   241   365   531   790   1328      23       375
-main  @21:04 (PROD)     408.2    402   253   406   535   684   1044       8       398
+run                            n   cum(s)  mean   p50   p90   p99  tail>700 body<=700 | TTFS p50 fast/30 | hops
+PROD  8bda7ce 20:38  #3213  1016    408.2   402   406   535   684        8      398   |    1039   6/30  |  3
+PROD  4a9d26b 22:20  #3186  1016    388.5   382   381   522   661        8      379   |     966   0/30  |  3
+PROD  32ac8e7 22:32  #3222  1016    440.4   433   422   575   808       28      417   |    1063   0/30  |  3
+prev  #3213 18:30           1016    386.0   380   378   515   743       12      374   |    1300   0/30  |  3
+prev  #3213 18:58           1016    386.8   381   376   517   731       15      374   |    1304   0/30  |  3
+prev  #3213 19:34           1016    416.7   410   409   554   736       16      404   |    1308  11/30  |  3
+prev  #3101 21:30           1016    392.4   386   365   531   790       23      375   |    1340   0/30  |  3
+prev  #3236 22:07           1016    419.2   413   389   559   835       19      389   |    1384   0/30  |  3
+prev  #3170 22:30           1019    367.2   360   353   484   672        8      352   |    1240   0/30  |  0
+--- real code changes, for scale ---
+prev  #3238 22:44           1019    413.4   406   400   547   715       15      396   |     586  17/30  |  0
+prev  repl-det 22:15        1019   1387.4  1361  1379  2226  3276      799      450   |    1247   0/30  |  0
 ```
 
-`tail>700` is the count of samples above 700 ms; `body<=700` is the mean of
-everything at or below it.
-
-### TTFS (step scenario, n = 30) and queue-hops
+Summary statistics over the null sets:
 
 ```
-run                  fast<600ms    min    p50   | hops  values (ms)
-#3213 @18:30 (prev)        0/30   1267   1300   |    3  [2270, 3013, 3446]
-#3213 @18:58 (prev)        0/30   1275   1304   |    3  [2256, 3107, 3125]
-#3213 @19:34 (prev)       11/30    273   1308   |    3  [1545, 2125, 2632]
-#3101 @21:30 (prev)        0/30   1308   1340   |    3  [2160, 3108, 3168]
-main  @21:04 (PROD)        6/30    210   1039   |    3  [1628, 2384, 2995]
+PREVIEW, runtime-identical (n=6)
+  cumulative : min 367.2  max 419.2  mean 394.7  sd 19.9  (CV 5.0%)  range 13.2%
+  mean       : min 360.3  max 412.6  mean 388.3  sd 19.9  (CV 5.1%)  range 13.5%
+PRODUCTION main-to-main (n=3)
+  cumulative : min 388.5  max 440.4  mean 412.4  sd 26.2  (CV 6.3%)  range 12.6%
 ```
 
-`fast<600ms` counts samples in the fast TTFS mode (see finding 3).
-
 ## Findings
 
 ### 1. The measurement is precise; the deployments are not
 
 The 18:30 and 18:58 runs — separate preview deployments, separate 1020-step
 executions — agree to **0.2%** on cumulative inline time (386.0 s vs 386.8 s)
-and **0.3%** on body mean (374 vs 374). A 1016-sample run is a stable
-instrument.
+and **0.3%** on body mean. A 1016-sample run is a stable instrument.
+
+So the between-run differences are not sampling noise. They are real differences
+between the deployments being measured. Some deployments land slow and stay slow
+for the whole run: the 19:34 run sits to the right at *every* quantile (p10 257
+vs ~236, p50 409 vs ~377), not just in the tail.
+
+### 2. Production is no more stable than preview
+
+The `main`-to-`main` comparison, which controls for the environment entirely:
+
+```
+20:38  8bda7ce  408.2 s
+22:20  4a9d26b  388.5 s
+22:32  32ac8e7  440.4 s
+```
 
-So the between-run differences are not sampling noise. They are real
-differences between the deployments being measured.
+The last two are **twelve minutes apart** and differ only by #3222, a Biome
+lint-fix commit with no semantic change — yet they are **13% apart**. The
+production null spread (12.6%) matches the preview one (13.2%).
 
-### 2. One deployment was simply slower, across its whole distribution
+This is the cleanest available demonstration that the variance is a property of
+the deployment/infrastructure draw, not of the environment tier or of anything
+in the diff.
 
-The 19:34 run sits to the right at every quantile: p10 257 vs ~236, p50 409 vs
-~377, body mean 404 vs ~374. That is +8%, and it is not a tail artifact — the
-entire body moved.
+### 3. The queue-hop tag has a blind spot: warm-process reuse
 
-This is the single largest source of inline variance, and nothing about the
-workload explains it. Some deployments land slow and stay slow for the whole
-run. Suspects (unconfirmed): instance placement, neighbour load on shared
-infrastructure, pool warmth at allocation time.
+`97_bench.ts` tags a step `queue-hop` when it is the first step body executed in
+its **process** (module-global `hasExecutedStepInProcess`). But Vercel can serve
+the follow-up invocation on the *same warm instance*, in which case the module
+global is already `true` and a genuine hop is tagged `inline`.
 
-### 3. Body and tail are separate signals that move independently
+The evidence: runs reporting **zero** hops still contain exactly three
+multi-second samples in the inline population, and a run reporting three hops
+contained two more:
+
+```
+#3170  (0 hops reported):  ... 750, 791, 792, 1630, 2851, 3408
+#3238  (0 hops reported):  ... 812, 819, 857, 1700, 2503, 2589
+32ac8e7 (3 hops reported): ... 1152, 1216, 1318, 1346, 2064, 3663
+```
+
+Counting reported hops plus inline samples above 1500 ms, the **actual** number
+of invocation boundaries per run ranges 3-7 while the reported count ranges 0-3:
+
+| Run | reported hops | inline samples >1.5 s | actual boundaries |
+|---|---:|---:|---:|
+| PROD 8bda7ce | 3 | 0 | 3 |
+| PROD 4a9d26b | 3 | 0 | 3 |
+| PROD 32ac8e7 | 3 | 2 (2064, 3663) | 5 |
+| #3213 ×3, #3101 | 3 | 0 | 3 |
+| #3236 | 3 | 4 (4263-4752) | 7 |
+| #3170 | **0** | 3 (1630, 2851, 3408) | 3 |
+
+Consequences:
+
+- The inline population is contaminated by 0-4 multi-second samples,
+  unpredictably. This is part of what I previously attributed to unexplained
+  infrastructure jitter.
+- `32ac8e7`'s apparent 13% production regression is partly this: it carries two
+  mislabeled hops worth ~5.7 s. Excluding samples >1.5 s, its mean drops
+  433 → 429 and cumulative 440.4 → 434.6 — smaller, but the bulk of its gap is
+  still a genuinely slow deployment.
+- Removing suspected hops does **not** tighten the null spread (14.7% vs 13.5%),
+  so mislabeling is a second, independent contaminant rather than the main
+  driver.
+
+No clean fix is available in the workflow alone. `COMPUTE_INSTANCE_ID`
+(`packages/core/src/runtime/compute-instance.ts`) has exactly the same blind
+spot by construction — its own docstring notes it is "shared by every invocation
+it handles." The only invocation-scoped identifier in the tree is
+`requestIdStorage` (keyed on `x-vercel-id`) in `packages/world-vercel/src/queue.ts`,
+which step bodies cannot currently see. See open item 1.
+
+### 4. Body and tail are separate signals that move independently
 
 For PR #3101 vs the `main` baseline, the comment flagged `P99 +15% 🔻` — but:
 
@@ -109,87 +186,110 @@ For PR #3101 vs the `main` baseline, the comment flagged `P99 +15% 🔻` — but
 - tail (>700 ms): 23 samples vs 8, total 19.7 s vs 6.7 s
 - cumulative: −4%, which is a −29 s body improvement plus a +13 s tail regression
 
-**The 🔻 was driven by 15 samples out of 1016.** Across the preview cohort the
-tail count ranges 12-23 and prod's 8 was the cohort minimum, so the flag was
-"fattest preview tail vs thinnest production tail," not a change in step cost.
+**The 🔻 was driven by 15 samples out of 1016.** Across the full cohort the tail
+count ranges 8-28, so the flag was "fat tail vs the cohort-minimum tail," not a
+change in step cost.
 
-Whether the tail is concentrated late in the run (growing event log) or
-scattered is **currently untestable** — see open item 2.
+Whether the remaining tail is concentrated late in the run (growing event log)
+or scattered is **still untestable** — see open item 2.
 
-### 4. TTFS is bimodal, and mode occupancy is a lottery
+### 5. TTFS is bimodal, and the environments differ
 
-TTFS samples fall into a fast mode (~200-400 ms, the in-process fast path) and a
-slow mode (~1300 ms). How many of a run's 30 samples land in the fast mode is
-close to random: 0, 0, **11**, 0 across previews and 6 on production.
+Mode occupancy is close to random. How many of a run's 30 TTFS samples land in
+the fast mode (~200-400 ms, the in-process fast path) rather than the slow one
+(~1300 ms):
+
+```
+production: 6/30,  0/30,  0/30
+preview:    0/30,  0/30, 11/30, 0/30, 0/30, 0/30   (and 17/30 on #3238)
+```
 
-Because **Best is a min over 30 samples**, a single lucky sample moves it 5×.
-That is the mechanism behind the `+523% 🔻` on a no-op change. Best is not a
-usable regression signal in its current form; the fast-mode *fraction* would be.
+Because **Best is a min over 30 samples**, one lucky sample moves it 5×. That is
+the mechanism behind `+523% 🔻` on a no-op change. Best is not a usable
+regression signal in its current form; the fast-mode *fraction* would be.
 
-### 5. Preview vs production is a real bias — but only for TTFS median
+Median TTFS, by contrast, is stable within an environment and **systematically
+different between them**:
 
 ```
-TTFS(step) p50:  preview 1300, 1304, 1308, 1340   |   production 1039
+TTFS(step) p50:  production 966, 1039, 1063   |   preview 1240, 1300, 1304, 1308, 1340, 1384
 ```
 
-The four preview medians agree within 3%; production is **20% faster than all
-of them**. Since `main` runs against production
+Since `main` runs against production and every PR runs against preview
 (`environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'preview' }}`,
-`.github/workflows/benchmarks.yml`) and every PR runs against preview, **every
-PR carries a built-in ~+25% TTFS "regression" that is pure environment
-mismatch**.
-
-### 6. The queue-hop count is stable
-
-Every run crossed exactly 3 invocation boundaries — the ~120 s per-invocation
-ceiling and the total run duration are consistent. Only the per-hop cost varies
-(1545-3446 ms), and with n=3 per run those percentiles are degenerate: p75, p90
-and p99 all collapse to the max sample.
+`.github/workflows/benchmarks.yml`), **every PR carries a built-in ~25% TTFS
+"regression" that is pure environment mismatch.** This is the one finding where
+the environment split genuinely matters.
 
 ## Hypotheses this data refuted
 
 Recorded so they are not re-proposed:
 
 - **"Preview vs production explains the inline STSO differences."** It does not.
-  Production (408.2 s) sits *inside* the preview range (386-417 s); one preview
-  run was slower than production. The environment split is a real bias for TTFS
-  p50 only.
-- **"The TTFS fast mode is a production-only effect."** No — the 19:34 preview
-  run hit it 11 times out of 30, more often than production did.
-- **"WO is an independent whole-run signal."** It is not. `WO − Σ(all STSO gaps)`
-  is 1019 ms (main) and 1362 ms (this) — WO is the gap total plus step-body
-  time, so it carries nothing the STSO rows do not already.
+  Production spans 388-440 s and preview 367-419 s — overlapping ranges, with
+  the single slowest run of all twelve being a production run.
+- **"The TTFS fast mode is a production-only effect."** No — a preview run hit it
+  11/30, more often than any production run, and #3238 hit it 17/30.
+- **"WO is an independent whole-run signal."** No. `WO − Σ(all STSO gaps)` is
+  ~1.0-1.4 s; WO is the gap total plus step-body time, carrying nothing the STSO
+  rows do not already.
+- **"±8% is the inline detection floor."** That was measured on the first four
+  runs available and understated it. With six runtime-identical preview runs the
+  observed range is 13.2%, and production is the same. Use ±10% as a floor and
+  expect occasional 13% excursions.
 
 ## Open items
 
-1. **Baseline against a preview deployment of `main`, not production.** Removes
+1. **Tag queue-hops by invocation, not process** (finding 3). Requires exposing
+   an invocation-scoped id to step bodies — `requestIdStorage` in
+   `packages/world-vercel/src/queue.ts` already tracks one per request; nothing
+   surfaces it to user code. Highest value: without it, the inline population is
+   silently contaminated and the queue-hop count is wrong on an unpredictable
+   subset of runs.
+2. **Keep `raw` in run order.** `computeStats` stores `raw: sorted`, which
+   destroys ordering at write time, so "is the slow tail concentrated in the last
+   200 steps?" — the obvious event-log-growth hypothesis — cannot be tested from
+   artifacts. Fix: store insertion order, sort locally inside `computeStats` for
+   the percentiles. No output changes. Would also let a mislabeled hop be
+   identified by position rather than by a >1.5 s heuristic.
+3. **Baseline against a preview deployment of `main`, not production.** Removes
    the built-in TTFS bias (finding 5). Requires `main`'s benchmark job to deploy
    and target its own preview rather than switching on `github.ref`.
-2. **Keep `raw` in run order.** `computeStats` currently stores `raw: sorted`,
-   which destroys ordering at write time, so "is the slow tail concentrated in
-   the last 200 steps?" — the obvious event-log-growth hypothesis — cannot be
-   tested from artifacts. Fix: store insertion order, sort locally inside
-   `computeStats` for the percentiles. No output changes.
-3. **Loosen or replace the p99 threshold for inline STSO.** At ±8% null spread,
-   a ±15% 🔻 will keep crying wolf. Body mean or p50 is the tighter signal.
-4. **Report TTFS fast-mode fraction instead of (or beside) Best.** Best is a min
-   over 30 samples and therefore a lottery; the fraction is the real signal.
-5. **Fix commit provenance on PR runs.** `.github/workflows/benchmarks.yml` sets
+4. **Loosen or replace the p99 threshold for inline STSO.** At a 13% null range,
+   a ±15% 🔻 will keep crying wolf.
+5. **Report TTFS fast-mode fraction instead of (or beside) Best.**
+6. **Consider repeated runs per commit.** With sd ≈ 5%, three runs would put the
+   standard error near 3% and make sub-10% regressions detectable. Cost is ~7
+   min/run of CI.
+7. **Fix commit provenance on PR runs.** `.github/workflows/benchmarks.yml` sets
    `GITHUB_SHA` to the PR head sha for the benchmark step, and the job log
    confirms the step env holds it (`342f5a3a5…`) — but the written artifact
-   records the synthetic `refs/pull/N/merge` sha (`9dfa2b64…`) instead. Root
-   cause not yet identified. Harmless for numbers, wrong for traceability.
+   records the synthetic `refs/pull/N/merge` sha (`9dfa2b64…`). Root cause not
+   identified. Harmless for numbers, wrong for traceability.
 
 ## Reproducing this analysis
 
-Download the artifacts for any two runs (must be run from inside a clone):
+Download the artifacts for any set of runs (must be run from inside a clone):
 
 ```bash
 gh run download <RUN_ID> --pattern 'bench-results-*' --dir /tmp/bench/<label>
 ```
 
 Each `bench-results-*.json` carries the full sample arrays (`metrics[].raw`), so
-no Datadog access is needed for distribution work. Compare them with:
+no Datadog access is needed for distribution work. Useful run lists:
+
+```bash
+# production baselines
+gh run list --workflow benchmarks.yml --branch main --limit 10 \
+  --json databaseId,conclusion,createdAt,headSha
+# every recent run, to find runtime-identical PRs (docs-only, lint-only, no-op)
+gh run list --workflow benchmarks.yml --limit 30 \
+  --json databaseId,conclusion,createdAt,headBranch,headSha
+```
+
+Only runs after `8bda7ce` (2026-07-30 20:38 UTC) have the inline/queue-hop
+scenarios and `raw` arrays; earlier ones use the old step-index windows and
+cannot be compared against these.
 
 ```js
 // node -e '<this>'
@@ -212,15 +312,18 @@ const sum = (a) => a.reduce((x, y) => x + y, 0);
 const mean = (a) => sum(a) / a.length;
 
 for (const [name, dir] of [['A', '/tmp/bench/a'], ['B', '/tmp/bench/b']]) {
-  const a = row(load(dir)[0], 'stso', '1020 steps (inline)').raw;
+  const r = load(dir)[0];
+  const a = row(r, 'stso', '1020 steps (inline)').raw;
+  const hops = row(r, 'stso', '1020 steps (queue-hop)')?.raw ?? [];
   console.log(
     name,
     'cum', (sum(a) / 1000).toFixed(1) + 's',
     'mean', mean(a).toFixed(0),
     'p50', q(a, 0.5),
     'p99', q(a, 0.99),
-    'tail>700', a.filter((v) => v > 700).length,
-    'body<=700', mean(a.filter((v) => v <= 700)).toFixed(0)
+    'reported hops', hops.length,
+    // finding 3: hops onto a reused warm process land here instead
+    'suspected mislabeled hops', a.filter((v) => v > 1500).length
   );
 }
 ```
@@ -234,14 +337,15 @@ but anything order-dependent does not (open item 2).
 process as the step before it — pure framework overhead) or after a
 **queue-hop** (first step of a fresh process, paying queue dispatch, client
 reinit and event-log replay). The workflow tags each step itself via a
-process-global in `workbench/example/workflows/97_bench.ts`; the split is ground
-truth, not inferred from step index or trace timestamps.
+process-global in `workbench/example/workflows/97_bench.ts` — ground truth
+rather than inferred from step index or trace timestamps, but subject to the
+blind spot in finding 3.
 
 That split was introduced in [#3213](https://github.com/vercel/workflow/pull/3213)
-precisely because the previous step-index windows (`1-20` / `101-120` /
-`1001-1020`, 19 samples each) mixed the two populations: whether an invocation
-boundary happened to land inside a window moved that window's P99 by hundreds of
-percent between identical runs. The analysis above is what became visible once
-that noise source was removed.
+because the previous step-index windows (`1-20` / `101-120` / `1001-1020`, 19
+samples each) mixed the two populations: whether an invocation boundary happened
+to land inside a window moved that window's P99 by hundreds of percent between
+identical runs. The analysis above is what became visible once that noise source
+was removed.
 
 See `packages/core/e2e/benchmark.test.ts` for the full metric definitions.
diff --git a/packages/core/e2e/benchmark.test.ts b/packages/core/e2e/benchmark.test.ts
index b25269fca2..e264d0a8f1 100644
--- a/packages/core/e2e/benchmark.test.ts
+++ b/packages/core/e2e/benchmark.test.ts
@@ -4,8 +4,9 @@
  *
  * Before reading a delta in the PR comment as a regression, see
  * BENCHMARK_VARIANCE.md in this directory: it measures the run-to-run noise
- * floor of each metric against real CI data (inline STSO ±8%, TTFS Best not
- * usable at all), and records which apparent signals are artifacts.
+ * floor of each metric against real CI data (inline STSO ±10%, TTFS Best not
+ * usable at all), and records which apparent signals are artifacts — including
+ * a blind spot in the inline/queue-hop split itself.
  *
  * Every run is triggered through an in-deployment route (`POST /api/bench` on
  * the workbench app) rather than by calling `start()` from this CI process. The
diff --git a/workbench/example/workflows/97_bench.ts b/workbench/example/workflows/97_bench.ts
index 3572eaf679..987528bdd4 100644
--- a/workbench/example/workflows/97_bench.ts
+++ b/workbench/example/workflows/97_bench.ts
@@ -46,6 +46,15 @@ export interface BenchStepTiming {
 // or redispatch via the queue after the prior invocation's ~duration limit)
 // resets this to false, so the first step body it runs is tagged
 // 'queue-hop'; every step after that in the same warm process is 'inline'.
+//
+// KNOWN BLIND SPOT: this detects *process* identity, not *invocation*
+// identity. When Vercel serves the follow-up invocation on the same warm
+// instance, this global is already true and a genuine queue-hop is tagged
+// 'inline' — leaving a multi-second sample in the inline population. Observed
+// on a real subset of CI runs (some report 0 hops while carrying three ~2-3s
+// inline samples); see BENCHMARK_VARIANCE.md finding 3 in packages/core/e2e.
+// Fixing it needs an invocation-scoped id exposed to step bodies;
+// COMPUTE_INSTANCE_ID has the same blind spot by construction.
 let hasExecutedStepInProcess = false;
 
 function stepKind(): 'inline' | 'queue-hop' {

From ea1a046ff2c582184570619e022a991a42069d99 Mon Sep 17 00:00:00 2001
From: "vercel[bot]" <35613825+vercel[bot]@users.noreply.github.com>
Date: Thu, 30 Jul 2026 23:25:44 +0000
Subject: [PATCH 3/3] Correct the boundary count: always 3, detection is
 all-or-nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The earlier "3-7 actual boundaries" was an artifact of treating every
inline sample over 1.5s as a mislabeled hop. The boundary count is not a
matter of inference: an invocation is capped at ~120s, so a run of
duration WO must cross ceil(WO/120s)-1 of them, which is exactly 3 for
every run in the set except the broken repl-det branch (11).

So the runs reporting 3 hops already found all of them, and their extra
multi-second inline samples are genuine stalls rather than mislabeled
hops. The >1.5s heuristic only finds hops on runs that reported none.

Detection turns out to be all-or-nothing — 3 of 3 or 0 of 3, never in
between, across eleven runs — which points at a per-run property rather
than a per-boundary coin flip. Also records the decisive test now that
#3186 persists the compute instance per step attempt: group one 0-hop
and one 3-hop run's attempts by instance id.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---
 packages/core/e2e/BENCHMARK_VARIANCE.md | 64 ++++++++++++++++---------
 1 file changed, 42 insertions(+), 22 deletions(-)

diff --git a/packages/core/e2e/BENCHMARK_VARIANCE.md b/packages/core/e2e/BENCHMARK_VARIANCE.md
index 7d8189ac7b..6563fdf945 100644
--- a/packages/core/e2e/BENCHMARK_VARIANCE.md
+++ b/packages/core/e2e/BENCHMARK_VARIANCE.md
@@ -32,7 +32,7 @@ anything.
 | inline STSO — cumulative / mean | sd ≈ 5% of mean; **observed range 13%** | treat ±10% as the floor |
 | inline STSO — p99 | 661-835 ms (±12%) | the ±15% 🔻 threshold is barely above noise |
 | inline STSO — tail count (>700 ms) | 8-28 samples | do not gate on it |
-| queue-hop STSO — count | **0-3 reported, 3-7 actual** | see finding 3; currently unreliable |
+| queue-hop STSO — count | **always 3 actual; 0 or 3 reported** | see finding 3; currently unreliable |
 | queue-hop STSO — per-hop cost | 1545-4663 ms (n≤3/run) | too few samples to read |
 | TTFS — p50 | ±3% within an environment | usable; see the environment bias in finding 5 |
 | TTFS — Best | 0-17 fast samples out of 30 | not a usable signal |
@@ -146,30 +146,43 @@ contained two more:
 32ac8e7 (3 hops reported): ... 1152, 1216, 1318, 1346, 2064, 3663
 ```
 
-Counting reported hops plus inline samples above 1500 ms, the **actual** number
-of invocation boundaries per run ranges 3-7 while the reported count ranges 0-3:
-
-| Run | reported hops | inline samples >1.5 s | actual boundaries |
-|---|---:|---:|---:|
-| PROD 8bda7ce | 3 | 0 | 3 |
-| PROD 4a9d26b | 3 | 0 | 3 |
-| PROD 32ac8e7 | 3 | 2 (2064, 3663) | 5 |
-| #3213 ×3, #3101 | 3 | 0 | 3 |
-| #3236 | 3 | 4 (4263-4752) | 7 |
-| #3170 | **0** | 3 (1630, 2851, 3408) | 3 |
+The number of boundaries is not a matter of opinion: an invocation is capped at
+~120 s, so a run of duration `WO` must cross `ceil(WO / 120s) - 1` of them. For
+every run here except the broken `repl-det` branch that is **exactly 3**.
+Detection, however, is all-or-nothing — 3 of 3, or 0 of 3, never in between:
+
+| Run | WO | boundaries required | reported hops | inline >1.5 s |
+|---|---:|---:|---:|---:|
+| PROD 8bda7ce | 416 s | 3 | 3 | 0 |
+| PROD 4a9d26b | 397 s | 3 | 3 | 0 |
+| PROD 32ac8e7 | 451 s | 3 | 3 | 2 (2064, 3663) |
+| #3213 ×3, #3101 | 396-423 s | 3 | 3 | 0 |
+| #3236 | 429 s | 3 | 3 | 4 (4263-4752) |
+| #3170 | 368 s | 3 | **0** | 3 (1630, 2851, 3408) |
+| #3238 | 414 s | 3 | **0** | 3 (1700, 2503, 2589) |
+| repl-det | 1389 s | **11** | **0** | 455 (branch is broken) |
+
+Note what this does *not* say. Where all 3 hops were detected, extra inline
+samples above 1.5 s are **genuine multi-second stalls, not mislabeled hops** —
+32ac8e7's two and #3236's four are real. The >1.5 s heuristic is only a reliable
+hop-finder on runs that reported none.
+
+The all-or-nothing pattern is itself informative: a per-boundary coin flip would
+produce 1-of-3 and 2-of-3 runs constantly, and none appear in eleven runs. It
+looks like a per-run property — whether that deployment had a warm instance
+available for the run's whole duration.
 
 Consequences:
 
-- The inline population is contaminated by 0-4 multi-second samples,
-  unpredictably. This is part of what I previously attributed to unexplained
-  infrastructure jitter.
-- `32ac8e7`'s apparent 13% production regression is partly this: it carries two
-  mislabeled hops worth ~5.7 s. Excluding samples >1.5 s, its mean drops
-  433 → 429 and cumulative 440.4 → 434.6 — smaller, but the bulk of its gap is
-  still a genuinely slow deployment.
-- Removing suspected hops does **not** tighten the null spread (14.7% vs 13.5%),
-  so mislabeling is a second, independent contaminant rather than the main
-  driver.
+- On affected runs the inline population absorbs three multi-second samples
+  (~1.6-3.4 s each, ~8 s total), and the queue-hop row vanishes entirely.
+- Removing all samples >1.5 s does **not** tighten the null spread (14.7% vs
+  13.5%), so mislabeling is a second, independent contaminant rather than the
+  main driver of run-to-run variance.
+- The hypothesis is warm-instance reuse but has **not been directly observed**.
+  It is consistent with the documented Fluid behavior, with the boundary
+  arithmetic, and with the magnitude of the orphaned samples — but the decisive
+  test is below.
 
 No clean fix is available in the workflow alone. `COMPUTE_INSTANCE_ID`
 (`packages/core/src/runtime/compute-instance.ts`) has exactly the same blind
@@ -178,6 +191,13 @@ it handles." The only invocation-scoped identifier in the tree is
 `requestIdStorage` (keyed on `x-vercel-id`) in `packages/world-vercel/src/queue.ts`,
 which step bodies cannot currently see. See open item 1.
 
+**How to confirm it directly.** #3186 (landed 22:20) persists the compute
+instance that ran each step attempt, and #2989 emits it as the OTEL
+`faas.instance` span attribute. Take one run that reported 3 hops and one that
+reported 0, and group their step attempts by instance id: warm reuse predicts
+the 0-hop run shows a single `cinst_` across all 1020 attempts while the 3-hop
+run shows four. That settles it without any new instrumentation.
+
 ### 4. Body and tail are separate signals that move independently
 
 For PR #3101 vs the `main` baseline, the comment flagged `P99 +15% 🔻` — but: