feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side - #3244
Conversation
🦋 Changeset detectedLatest commit: 82bd72e The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 424581ms → this run 423919ms (Δ -662ms, 0%) 1020 steps (queue-hop) Cumulative STSO time: main 6430ms → this run 9083ms (Δ +2653ms, +41%) 📜 Previous results (1)2308af2Fri, 31 Jul 2026 00:25:05 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
There was a problem hiding this comment.
Pull request overview
Adds a cross-environment “resilient start” guard by stamping the creator’s environment into queued runInput, wiring it through the v4 events meta path, and treating server rejections for environment mismatch as terminal (ack, don’t retry) to prevent the same wrun_ from being created in two different Vercel environments.
Changes:
- Introduces
World.getEnvironment?()and carries an optionalenvironmentfield throughRunInputandrun_startedevent data. - Implements environment resolution in
@workflow/world-vercelvia a sharedresolveClientEnvironment()used both forx-vercel-environmentandgetEnvironment(), and forwardsenvironmentthrough v4 frame meta. - Updates the core runtime to (a) stamp creator environment at
start(), (b) log deployment pin mismatches diagnostically, and (c) ack queue messages onrun_environment_mismatchrejections instead of retrying.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/queue.ts | Adds optional environment field to RunInputSchema. |
| packages/world/src/queue.test.ts | Adds schema coverage for environment round-trip and unknown-key stripping behavior. |
| packages/world/src/interfaces.ts | Adds optional World.getEnvironment?() hook contract. |
| packages/world/src/events.ts | Extends run_started event data with optional environment. |
| packages/world-vercel/src/utils.ts | Adds resolveClientEnvironment() and reuses it for x-vercel-environment header generation. |
| packages/world-vercel/src/utils.test.ts | Tests environment resolution matrix and header agreement. |
| packages/world-vercel/src/index.ts | Implements getEnvironment() for world-vercel using resolveClientEnvironment(). |
| packages/world-vercel/src/events.ts | Routes environment through v4 meta splitting allowlist. |
| packages/world-vercel/src/events.test.ts | Tests splitEventDataForV4 includes/omits environment appropriately. |
| packages/world-vercel/src/events-v4.ts | Adds environment to v4 frame meta and fixes error-code extraction to read error field. |
| packages/world-vercel/src/events-v4.test.ts | Tests error-field fallback and meta wire round-trip for environment. |
| packages/core/src/runtime/start.ts | Stamps world.getEnvironment?.() into queued runInput.environment. |
| packages/core/src/runtime/start.test.ts | Tests environment stamping/omission behavior. |
| packages/core/src/runtime/cross-environment-start.test.ts | Adds end-to-end runtime handler tests for deployment pin diagnostics + cross-env 422 ack behavior (including turbo behavior). |
| packages/core/src/runtime.ts | Adds deployment pin mismatch diagnostic + terminal handling for cross-env resilient start rejection. |
| packages/core/src/classify-error.ts | Adds isCrossEnvironmentStartRejection and constant for the server error code. |
| packages/core/src/classify-error.test.ts | Adds classification/negative tests for cross-environment rejection detection. |
| .changeset/world-vercel-get-environment.md | Changeset for @workflow/world-vercel additions/behavior changes. |
| .changeset/world-get-environment.md | Changeset for @workflow/world interface/schema additions. |
| .changeset/core-cross-environment-start-guard.md | Changeset for @workflow/core runtime behavior change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…environment queue deliveries client-side `start()` makes two writes that have to land in the same tenant: the `run_created` event, attributed to whatever environment the caller authenticates as, and the queue message, pinned to a deployment. A misconfigured caller can split them — writing the run to one environment while addressing the message to a deployment in another. The consumer finds no run under its own tenant, the backend's resilient start (`run_started` creates the run when `run_created` was never seen) mints a second copy of the same run id in the consumer's environment, and both copies are real: the creator's sits pending forever while the other executes. The deployment id is not the discriminator — it matched end to end in the incident that motivated this. The environment is. So carry it: add an optional `World.getEnvironment()`, implement it in world-vercel from the same resolution that produces the `x-vercel-environment` header, and stamp it into the queue message's `runInput`. The consuming deployment already knows its own environment, so it can refuse the delivery itself with no server coordination — and refuse before `run_started`, the write that would create the fork. The refusal acks the message instead of throwing: the mismatch is baked into the message, so every redelivery would reach the same verdict and throwing would hot-loop until MAX_QUEUE_DELIVERIES. Both sides must be known for the check to run, so worlds with a single tenant (local, Postgres) and runs started by an older SDK behave exactly as before. A companion diagnostic logs a deployment-id mismatch without refusing, since deployment ids differ for benign reasons too. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
3021bcb to
2308af2
Compare
pranaygp
changed the title
TooTallNate
left a comment
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed as the third layer of the incident stack (with #3243 and vercel/wait-for-deployment-action#6), and against the earlier investigation of this same race from June (workbench-express, same mechanism: prod URL + preview dpl_ from the changeset-release branch). This is the right hardening and the design choices hold up under scrutiny:
- Refusing before
run_startedis the only placement that prevents the fork rather than reporting it — verified the guard sits ahead of the event write and ahead of all attempt-dependent branching, with tests pinning that ordering. - Ack-and-discard over throwing is correct: the verdict is baked into the message, and it matches how the handler already treats
EntityConflictError/RunExpiredError. - Fail-safe skip matrix is genuinely fail-safe: I checked each row (no
getEnvironmenton local/postgres, absent field from older SDKs,undefinedfrom missing env vars) and none can produce a false refusal. Keeping producer stamp and consumer header derived from oneresolveClientEnvironmentis the load-bearing detail, and the agreement test locks it. - Wire compat claims check out:
run_startedeventData is built field-by-field soenvironmentnever reaches the server, and the unknown-key-stripping regression test protects the old-consumer direction. - The deployment-pinning check staying diagnostic-only is right, given
dpl_local@<version>churn.
On CI: the Biome failure is the same one inherited from main (organizeImports in files whose only change here is additive imports — the two flagged files fail identically on main), and the two e2e failures are flakes unrelated to the guard: the local-prod lane doesn't implement getEnvironment at all, and the Vercel lane failure is a clock-skew sleep assertion in a test whose run executed (i.e. the guard matched and passed).
Does this make #3243 / #6 irrelevant? No — worth being explicit since it's been asked. This PR removes the corruption (no more forked run IDs, no cross-tenant ack noise), but a misconfigured caller still ends up with a stuck-pending run and a red e2e lane — just a diagnosable one. #3243 is still needed to stop CI from producing the mismatched pair in the first place, and the long-term fix for the action remains the vercel/api one identified in the June investigation (publish the dashboard URL in the env-scoped GitHub Deployment status so ID resolution is tokenless and exact), which supersedes #6's token path.
Two non-blocking notes inline. Approving.
| if (projectConfig) { | ||
| return projectConfig.environment || 'production'; | ||
| } | ||
| return process.env.VERCEL_ENV || undefined; |
There was a problem hiding this comment.
Non-blocking question (custom environments): for a deployment in a Vercel custom environment, VERCEL_ENV reports preview and the custom environment's name lives in VERCEL_TARGET_ENV. Meanwhile the proxy path will happily stamp an arbitrary string from projectConfig.environment (e.g. staging). If the backend keys tenants on the custom-env name but this helper returns preview on the in-deployment path (or vice versa), the guard could refuse a legitimate delivery — the one failure mode this PR is otherwise careful to make impossible.
Given the interface doc's MUST ("match the attribution the backend will actually apply"), it's worth verifying what workflow-server actually attributes for (a) an OIDC client in a custom environment and (b) a proxy client sending a custom-env x-vercel-environment, and either incorporating VERCEL_TARGET_ENV here or documenting that custom environments are out of scope. Non-blocking because such a setup already forks today — the guard can only make it louder, not worse — but a wrong answer here turns into a false refusal rather than a skipped check.
There was a problem hiding this comment.
Verified against the actual minting code in vercel/api, and you were right to flag it — VERCEL_ENV alone was wrong for custom environments, in exactly the false-refusal direction:
- Runtime OIDC path: the deployment token's claim is minted as
environment: customEnvironment?.slug ?? envTarget(services/api-deployments/src/services/gen-oidc-token.tscaller increate-deployment.ts), so a custom-env deployment is attributed by its slug, notpreview— whileVERCEL_ENVin that same process sayspreview. - Proxy path:
x-vercel-environmentacceptsproduction,preview, or a custom environment's slug or ID, and the minted claim is alwaysmatchingEnv.slug(services/api-workflow/src/index.ts). - The fix (82bd72e): the in-deployment branch now returns
VERCEL_TARGET_ENV || VERCEL_ENV.VERCEL_TARGET_ENVis populated from exactly the same pair as the claim (customEnvironmentSlug || projectEnvTargetinpackages/projects-transforms/gen-system-envs.ts), so it matches the tenant key by construction, for standard and custom environments alike;VERCEL_ENVstays as the fallback whereVERCEL_TARGET_ENVisn't injected (e.g.vercel dev). Tests cover the custom-env case and the standard-env no-op.
One residual documented rather than solved: on the projectConfig path the proxy accepts a custom env ID but attributes the slug, so configuring the ID would stamp a value the consumer can't match. The helper's doc now says to configure the slug; canonicalizing an ID client-side would need an API call, which didn't seem worth it for that corner.
| return false; | ||
| } | ||
|
|
||
| runLogger.error( |
There was a problem hiding this comment.
Non-blocking (operational visibility): the refusal is logged only on the consuming deployment's runtime logs — the environment the operator is probably not looking at, since from their side the symptom is a run stuck pending in the creator's environment with no error anywhere on it. That's still strictly better than the fork, and I agree nothing more is possible client-side (the consumer can't write to the creator's tenant). Worth a follow-up though: this is exactly the case for the "carry the creator's environment in runInput so resilient start can detect/refuse server-side" hardening — the server can see both tenants and could mark the creator's run failed or surface a health annotation instead of leaving it pending forever.
There was a problem hiding this comment.
Agreed on all points — the consumer-side refusal makes the fork impossible but leaves the creator's run pending with the evidence in the other environment's logs. Tracking the server-side layer as follow-up work; its ingredients are already mapped out from this incident:
- workflow-server#689 (closed as superseded by this PR) contains a working implementation of the resilient-start guard and can be revived as the reporting/annotation layer you describe — the server is indeed the only place that can see both tenants and mark the creator's copy failed instead of pending-forever.
- Two prerequisites discovered along the way: the v4 events wire rebuilds eventData from field allowlists (
parseV4EventMeta/buildEventData), sorunInput.environmentneeds explicit plumbing on both sides before the server can see it; and covering old SDKs atrun_createdtime needs the deployment'stargetexposed from vercel/api'sget-deployment-infoendpoint (the Cosmos deployment doc it already loads carries it).
Until then, the creator-side symptom is at least the pre-existing one (stuck pending with "unhealthy deployment" health checks) rather than a silent fork, and the consumer-side error log names both environments and the runId for anyone who goes looking.
…_ENV For a deployment in a Vercel custom environment, the OIDC token's environment claim is the custom environment's slug (the platform mints `customEnvironment?.slug ?? envTarget`) while VERCEL_ENV reports 'preview' — so keying the cross-environment guard on VERCEL_ENV could false-refuse a legitimate delivery, e.g. a CLI client attributed to 'staging' starting a run on the staging deployment. VERCEL_TARGET_ENV is populated from exactly the same slug-or-target pair as the claim, so prefer it, keeping VERCEL_ENV as the fallback for contexts that don't inject it. Also sorts runtime.ts imports per the Biome rule that landed on main in #3241. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
|
No backport to This adds new public API surface and wire format rather than fixing a defect in existing To override, re-run the Backport to stable workflow manually via |
The bug
start()makes two writes that have to land in the same tenant:run_createdevent, attributed to whatever environment the caller authenticates as, andA misconfigured caller can split them — writing the run to one environment while addressing the message to a deployment in another. What happens next is not a failure, it is a fork:
run_startedcreates the run whenrun_createdwas never seen) mints a second copy of the same run id in the consumer's environment;This happened on 2026-07-30: a CI harness ran with
WORKFLOW_VERCEL_ENV=productionwhileVERCEL_DEPLOYMENT_IDpointed at a preview deployment, andwrun_41KYTFGXEC0GPWVN8C0J0KG9ZKended up existing in both environments of the same project, with two distinctrun_createdevents 1.18s apart.The deployment id is not the discriminator — it matched end to end in that incident. The environment is, and nothing on the wire carried it.
The fix
Carry the environment, then check it on the consumer.
@workflow/world— optionalWorld.getEnvironment(): string | undefined(sync, side-effect free, styled aftercreateRunId), and an optionalenvironmentonRunInputSchema.@workflow/world-vercel— implements it. Both auth paths, one source of truth: withprojectConfig(CLI/CI via the api.vercel.com proxy) the backend attributes the write to thex-vercel-environmentheader, sogetEnvironment()and that header now derive from one sharedresolveClientEnvironment()helper and cannot drift; withoutprojectConfig(inside a deployment, OIDC) it readsVERCEL_ENV, which the platform mints the token'senvironmentclaim from.@workflow/core—start()stamps the value into the queue message'srunInput, and the queue handler refuses a delivery whose creator environment disagrees with its own.Why the rejection is client-side
The consuming process already knows its own environment, so it needs nothing from the server to catch this. More importantly it can refuse before
run_started— the write that would create the fork. Any check that happens after that write is too late; refusing before it means the resilient start never creates anything, and the creator's run is left pending in its own environment where a human can find it.Why it acks instead of throwing
The mismatch is baked into this message: it is pinned to this deployment, and the creator's environment is a field in the payload. Every redelivery would reach the identical verdict, so throwing would hot-loop the handler until
MAX_QUEUE_DELIVERIESand end with a misleading max-deliveries failure. Returning normally acks and discards, which is how the handler already treats deliveries whose verdict cannot change (EntityConflictError,RunExpiredErroron therun_startedpath both log and return).The error log names both environments, the run id, the pinned deployment, and the likely cause (
WORKFLOW_VERCEL_ENVfor CLI/CI clients, or the OIDC token's environment inside a deployment).Fails safe, by construction
The check runs only when both sides are known, so nothing that works today changes behavior:
runInput.environmentworld.getEnvironment()undefinedproductionproductionproductionpreviewundefinedis deliberately preserved rather than defaulted: guessing'production'in a bare Node process would fabricate a mismatch against a genuine preview deployment. A wrong answer here is worse than no answer.Secondary: deployment-pinning diagnostic
Alongside it, when
runInput.deploymentIdand the handler's owngetDeploymentId()are both known and differ, the handler logs an error with both ids and the run id — and continues.Warn rather than refuse, because a differing deployment id is not by itself evidence of the fork.
world-localderives its id from the installed package version (dpl_local@<version>), so upgrading the SDK mid-run changes it with nothing wrong; refusing on that signal would strand correct runs. The two checks live side by side and are individually tested, with the environment guard short-circuiting first so a cross-environment delivery produces one clear error rather than two.Wire compatibility
Verified read-only against workflow-server and the SDK's own schemas.
run_startedeventData is built field-by-field inruntime.ts, not spread fromrunInput, soenvironmentnever reaches the events API — it exists only in the queue payload. It would have been safe anyway:CreateEventSchemaV2.eventDataisz.record(z.any())andInitialRunAttributesSchemais.passthrough(), so an unknown key is accepted, not rejected.RunInputSchema,WorkflowInvokePayloadSchema, andQueuePayloadSchemaare all plainz.object— zod strips unknown keys instead of rejecting them, so a message from a new producer is consumed normally by an older deployment mid-rollout (it just skips the check). Pinned by a regression test so nobody makes these strict later.No deviation from the intended design was needed.
Tests
packages/world/src/queue.test.ts—RunInputSchemaround-trip, absent field, non-string rejection, unknown-key stripping.packages/world-vercel/src/utils.test.ts—resolveClientEnvironmentmatrix, plus an agreement test asserting it always equals thex-vercel-environmentheader for the same config, so the two can't drift.packages/core/src/runtime/start.test.ts— stamps the environment, omits it whenundefined, omits it when the world doesn't implement the hook.packages/core/src/runtime/cross-environment-delivery.test.ts— mismatch produces no events at all (norun_started), acks with 204, logs both environments; refuses on a redelivery too; matching environments execute normally; each skip case (field absent, hook absent, hook returnsundefined, re-enqueued message) executes normally; plus the deployment-pinning diagnostic and its ordering against the guard.pnpm typecheckpasses (41/41 packages).@workflow/core1671 passed + 3 expected-fail,@workflow/world84 passed,@workflow/world-vercel322 passed.pnpm lintreports 2 pre-existing errors onmain(nestedbiome.jsonfiles declaring$schema2.0.0 against CLI 2.4.4) — confirmed present onorigin/mainwith these changes stashed, and untouched here.🤖 Generated with Claude Code