Slot event identity (SDK 3/3): number events by slot in the Local and Postgres Worlds - #3246
Draft
VaguelySerious wants to merge 20 commits into
Draft
Slot event identity (SDK 3/3): number events by slot in the Local and Postgres Worlds#3246VaguelySerious wants to merge 20 commits into
VaguelySerious wants to merge 20 commits into
Conversation
🦋 Changeset detectedLatest commit: 7dd9863 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (1422 failed)astro (123 failed):
example (123 failed):
express (123 failed):
fastify (123 failed):
hono (123 failed):
nextjs-turbopack (148 failed):
nextjs-webpack (148 failed):
nitro (123 failed):
nuxt (123 failed):
sveltekit (142 failed):
vite (123 failed):
📋 Other (246 failed)e2e-vercel-prod-nest (123 failed):
e2e-vercel-prod-tanstack-start (123 failed):
vercel-multi-region (27 failed)nextjs-turbopack (27 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 📋 Other
❌ vercel-multi-region
|
VaguelySerious
added
the
event-log-race-repro
VaguelySerious
added
event-log-race-repro
…s its start The synthetic `step_created` a lazy step start materializes was written with a plain create-if-absent write, so losing its position surfaced as a duplicate- entity error - which the runtime reads as "another handler owns this step" and skips, leaving the step claimed but never run. It now takes the position the same way every other event does, and a loss is reported as a slot conflict the caller can merge and re-propose.
An optimistic start runs against a locally synthesized run row, so the workflow start time this invocation reports comes from the client clock while every later replay reads the persisted run. Sending that instant as the run_started event's occurredAt lets a backend record it as the run's startedAt, so the two agree instead of differing by the round-trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rt time `createdAt` is stamped when a write begins, before its final slot is known, so it disagrees with slot order in two ways: a writer that loses a slot re-proposes above the winner while keeping its earlier stamp, and a caller that reserves a range of slots for one flush commits them in whatever order the network returns. Replay consumes the log in list order and never sorts, so listing by `createdAt` hands it an order no execution produced. Slot events now report one shared order time and let the existing event-id tie-break do the ordering, matching the Postgres World's `orderBy(eventId)` and the Vercel World's sort key. ULID runs keep their wall-clock order, which their ids agree with anyway.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| * returns. A ULID-numbered run keeps its wall-clock order, which its ids agree | ||
| * with anyway. | ||
| */ | ||
| const eventOrderTime = (event: { eventId: string; createdAt: Date }): number => |
Contributor
There was a problem hiding this comment.
Suggested change
| const eventOrderTime = (event: { eventId: string; createdAt: Date }): number => | |
| const eventOrderTime = <TEvent extends { eventId: string; createdAt: Date }>( | |
| event: TEvent | |
| ): number => |
Non-generic eventOrderTime parameter type collapses the generic T of paginatedFileSystemQuery, causing a TypeScript build break across event-typed call sites in @workflow/world-local.
Two independent sources of `CorruptedEventLogError` on well-formed event logs, both found by dumping the logs of runs that failed that way. A hook delivery is ordered by when its event row commits, not by when the payload arrived, so a delivery that races a disposal — arriving first, committing second — lands after its own `hook_disposed` in the log. The hook's consumer retired on the disposal, so nothing consumed that event on any replay: a divergence that recurs identically every attempt and escalates to a terminal error. The consumer now stays registered as a tombstone and discards the late delivery, which is what `disposeHook` already assumed when it settled every awaiter. Separately, the unconsumed-event check gave the VM a flat 100ms of wall clock to register the next event's consumer. Real logs routinely need more: a hook payload fanning out into steps measures 254-717ms between the delivery and the first `step_created` it causes. The check now re-arms while a delivery is still in flight — the condition `scheduleWhenIdle` already polls — bounded by `WORKFLOW_DEFERRED_CHECK_MAX_GRACE_MS` so an abandoned delivery cannot park it forever.
Contiguous allocation is not the same thing as a gap-free published log: a slot claimed by an operation that then fails for a reason of its own is never filled, and once a later slot is published that gap is permanent. What the scheme actually buys is explicit contention and a log that reads in write order. Nothing consumed the proof, so this is a comment and docs correction.
…e waited for Three diagnostic gaps that together made replay divergence unreadable: - `composeLogLine` dropped `errorMessage` whenever the message did not already contain it, so a warn carrying an error alongside its own summary line logged the symptom and none of the diagnosis. - An unconsumable event named only itself. It is almost always an event whose entity this replay never issued, so the pending invocation queue is what distinguishes "never issued" from "issued under another id". - An inline step batch abandoned on a fenced claim logged neither which member was fenced nor how the others settled. The fence is per-write, so a batch can split: the rejected claim writes nothing while a sibling on a different slot commits. Also read a failed run's error through `returnValue()` in the race-repro harness — `runs.get` returns the raw serialized payload, so every corruption in the report carried a code and no message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…atch An inline step's step_started claim is fenced per-write under slot identity, so a 409 only proves another writer took that write's number — routinely true, since the backend allocates outside events from the same next-free pointer the client reserves from. Abandoning the whole batch on it left the loser's events landing seconds later, after a whole later phase, in an order no single replay could consume. stepClaimFence keeps a watermark-guarded run on its single shared fence (a 412 does mean the view is stale, and the batch is meant to fail as a unit) and gives a slot-numbered run an in-place reclaim: merge the delta, reserve past it, re-claim. The reservation pointer is now absolute and only moves forward, so a merge cannot hand the retrying writer a slot a sibling is still in flight on.
Contributor
📊 Workflow Benchmarks❌ The benchmark run for commit Backend:
📜 Previous results (1)a67b345Fri, 31 Jul 2026 21:05:15 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Contributor
Event Log Race ReproNo event-log regressions in the latest repro job. 1400 runs hit harness-side Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
Showing 20 of 1400 non-completed runs. The |
A slot names a position in the replay order, so allocation has to hand out a position no published event sits above. Handing out the lowest free position instead let a late `step_completed` drop into a hole beneath its own `step_created`/`step_started`, and every replay of that run then met a completion for a step it had not started. The book now keeps a monotonic ceiling: a reservation goes above every position the book has ever seen, and releasing one does not lower it. `run_created` takes the first slot outright rather than allocating it, since a `run_started` racing it can already have moved the book past that position. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…event A divergence is a disagreement between the order the log records and the order a replay reconstructs it in, so the log's own ordering is the only evidence that separates the candidate causes. The runs live on an ephemeral preview deployment, so the window has to be read while the job is still running. Sampled (6 runs) and kept out of the sticky comment, which has a size limit; the results artifact carries them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3234. Review that first — this PR only makes sense once the client can claim a slot.
What
Slot identity is only useful if a World can keep it, so both first-party Worlds now allocate, honour, and defend dense per-run positions.
evnt_…001is a run's first event,evnt_…002its second, with no gaps — so the highest number is the event count, and a reader can prove its copy of a log is complete.Behaviour is opt-in per run via
WORKFLOW_SLOT_IDENTITY, and a run keeps the scheme it was created with for life.How
SPEC_VERSION_MAX_SUPPORTEDseparates the newest version a World can read from the version it stamps. Without the split, the flag that turns slot identity on for new runs would make every World reject the runs it had just created. Worlds opt into minting individually, through thespecVersionthey declare —mintedSpecVersion()is the one place that reads the flag.(run_id, id), migration0018), because under slot identityevnt_…001exists once per run. That key is then the authority: a unique violation means "this position is taken", and the writer re-probes and tries the next free one. Step ids get the same treatment for the same reason. The run leads both keys, so the existing run-scoped range scans stay a single index seek — which also makes the standalonerun_idindexes redundant.max++, is what guarantees progress: each round at least one writer wins.SlotConflictError) carrying the events the caller was missing, and nothing is materialized for it — a step row left behind by a losing attempt would make the caller's re-post trip its own orphan and read that as "a concurrent handler won the create".assertWorldSupportsRuntimeProtocolaccepts a World declaring anything fromSPEC_VERSION_CURRENTup toSPEC_VERSION_MAX_SUPPORTED, and a turbo invocation seeds its slot floor from the backgroundedrun_startedit never sees in its snapshot.SPEC_VERSION_CURRENTstays 5 — nothing mints spec 6 unless the flag is set.Testing
packages/world-postgres/test/slot-identity.test.ts(19 tests, Postgres testcontainer) andpackages/world-local/src/storage/slot-identity.test.ts: dense numbering, both mode-pinning 400s, the conflict delta, and the no-orphan guarantee.1…6with no holes.world-postgres181,world-local522,world98,core1687.Follow-ups
events.listByCorrelationIdis not yet run-scoped; a correlation id is only unique within its run under slot identity. Needs arunIdonListEventsByCorrelationIdParams, so it is deliberately out of this PR.world-vercelis the remaining World; it lands with the backend stack.🤖 Generated with Claude Code