🤖 perf: replay synthetic session tapes through the desktop app with egress blocked (T3) - #5612
Conversation
…only replay mode XUM_REPLAY_TAPES maps workspace ids to tapes; onChat for a mapped workspace is served from the tape (desktop only, after the renderer egress block), and the whole process refuses sends, history changes and provider model creation. Includes the #5587 read-side fixes and the Start Here refusal dialog. Signed-off-by: Thomas Kosiewski <tk@coder.com>
…re and make perf-tape-replay Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…mode Signed-off-by: Thomas Kosiewski <tk@coder.com>
…renderer replay paths Signed-off-by: Thomas Kosiewski <tk@coder.com>
…import path Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…Coder CLI probe in perf.tapeReplay strace of the scenario showed Chromium's spellcheck dictionary download (browser-internal, outside webRequest) and the backend's coder whoami probe reaching the network.
The refused-Start-Here test added clickStartHere/findEnabledOkButton but left the original test with an inline copy of the same lookups. Use the helpers in both tests; behavior is unchanged. Signed-off-by: Thomas Kosiewski <tk@coder.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
Security findingsAdvisory findings (1)
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4de6de0566
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: 4de6de0566
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
…host DNS, non-retryable refusal)
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…r and git remotes in perf.tapeReplay Codex security round 1: browser/CLI clients of the API server would render tapes outside the desktop egress block. UAT round 1: the app's background git remote queries and gh calls reached the network; the scenario now shadows coder/gh with failing logged stubs and wraps git with GIT_ALLOW_PROTOCOL=file.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51e70a96d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s play, task admission disposed)
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9fc75d848
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c0f38ebc7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…only file URLs, documented read-only funnels)
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad52106e6a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8289e2817
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Stopping the review loop: the 6-round Codex cap for this PR is used up. This PR is not ready.
The next step needs a decision from the owner: allow one more Codex round (fixes, validation, remote UAT on the new head), or accept the findings as follow-ups. |
…rts in the child test, shared timer limit, drop fixture copy test)
…side the perf harness (round 7)
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Final state of the round-7 exception round. Head:
This PR is ready for the conductor. I did not merge or enqueue it. |
|
Readiness record: merging at
Generated with |
|
Re-enqueueing once. The merge queue removed this PR because |
…filing Xum page (coder#5625) ## Summary Documents two merged perf features on the Profiling Xum page (`docs/reference/profiling.mdx`), and fixes one wrong runtime message. 1. **Report slowness** (coder#5558): a new `## Report slowness` section. - How to run it: the palette command, a link to its shortcut on the keyboard shortcuts page, and `xum api perf-reports create`. - Where the report goes: `~/.xum/perf/reports/<id>/`. - A table of every file in the report: `snapshot.json`, `trace.json`, `environment.json`, `README.txt`, `captures/` with `manifest.json`, `hangs.json` and `app-metrics.json`. The last two are desktop only. - What it never contains: tapes, chat content, prompts, tool payloads or environment variables. - A warning that path redaction is partial, not anonymization: copied CPU profiles keep the folder names below `~` (coder#5561). - The "Report a performance problem" checklist now starts with this report. - `docs/config/keybinds.mdx` keeps the shortcut and links here, so each fact has one home. 2. **Tape replay** (coder#5612): a new "Replay a synthetic tape (contributors)" subsection. - What `make perf-tape-replay` runs and where its output goes. - It replays only synthetic fixtures. A warning says never to replay or commit real tapes. - An accordion explains how replay stays offline. Replay runs only when `XUM_E2E=1` and `XUM_REPLAY_HARNESS=1` are set, and the test harness sets them, not the Makefile. The app blocks renderer requests, while the harness takes background network offline. Tape paths are checked for absolute local syntax only: `..` and symlinks are accepted. 3. **Refusal message** (`sessionTapeReplaySource.ts`): it now says "Run make perf-tape-replay to use the isolated replay harness." The old text claimed the make target sets the markers. The existing test matches only `/only inside the perf harness/`, which is unchanged, so this PR adds no exact-copy test. ## Validation - I ran the documented commands against a sandbox `xum server` with a temporary `XUM_ROOT`: - `perf-reports create` with the experiment off fails with `PRECONDITION_FAILED` and the documented message. - With the experiment on, it writes the documented layout: `README.txt`, `captures/` with `manifest.json`, `environment.json`, `snapshot.json` and `trace.json`. Folders are 0700 and files 0600. - The `environment.json` keys match the page, and the home path appears nowhere in the bundle. - Two concurrent runs get `CONFLICT` with "a slowness report is already being written". - I rendered the page locally with the CI-pinned Mintlify and checked the new sections at desktop and 390 px widths. Both accordions expand. - `make static-check-full` passes, including `mintlify broken-links`. `sessionTapeReplaySource.test.ts` passes (25 tests).    --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$30.02`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=30.02 -->
1 participant
Summary
Adds
perf.tapeReplay. It is an opt-in harness mode (XUM_REPLAY_TAPES) in which the desktop app serves a session tape as a workspace'sworkspace.onChatstream. The replay runs through the real backend subscription, MessagePort oRPC,WorkspaceStoreand React. A Playwright perf scenario replays a committed synthetic tape, checks the rendered transcript, and writes timings like the other perf specs.make perf-tape-replayruns it.Background
T1 (#5529) records tapes. T2 (#5568) added the offline loader and replay driver. T2's app integration was removed because the real renderer fetches remote URLs found in recorded content (markdown images). This PR adds app playback together with network isolation. It also picks up the read-side items from #5587 that matter now that the app reads tapes.
Implementation
XUM_REPLAY_TAPES='{"<workspaceId>":"<absolute .jsonl>"}'. Unset or blank: no work beyond one env lookup. Set (even unparseable): the whole process is in replay mode. Tapes are served only inside the perf harness, which also setsXUM_E2E=1andXUM_REPLAY_HARNESS=1. Without those markers every mapped workspace shows the refusal row and nothing is replayed. Tape paths must be absolute local paths (allowlist: POSIX/...or a Windows drive path). UNC shares,\\?\and\\.\prefixes, URLs and relative paths are refused before any read.gh, Coder CLI probes) is not tied to tape content. The harness isolates it (see Scenario below).session_tape_replaysend error that namesXUM_REPLAY_TAPES. A task admission token passed with a send is disposed. Provider model creation refuses at its three chokepoints (ProviderModelFactory.createModelEffect,createEvaluationModelEffect,evaluationModelFactory.createEvaluationModel), so background callers (status, title, compaction, 🤖 fix: XUM_MOCK_AI still sends background claude-haiku requests #5604) cannot reach a provider. Telemetry is disabled. Other writes, such as plan-review snapshots, are documented as unguarded: replay maps scratch workspaces only.sessionTapeReplaySource.ts).subscribeWorkspaceChatbranches before any session access for a mapped workspace. It serves fresh full subscriptions only, and refusessinceandlive("reload to replay"). It acceptsclosedandstoppedtapes and refuses rejected and truncated ones. It checks the headerworkspaceIdHashand requires exactly one successfulcaught-up. Events play at recorded offsets. The only change to an event:caught-up.hasOlderHistoryis set to false, so the client never pages in live history. The stream then stays open with transport heartbeats. Refusals are typed (ORPCError+SESSION_TAPE_REPLAY_REFUSAL_DATA). WorkspaceStore shows them as a non-retryable "Session tape replay refused" row.webRequest.onBeforeRequeston the default session. Localfile:URLs (no host, no UNC path),data:,blob:anddevtools:pass. In dev-server mode, the exact app-page and terminal-page dev origins also pass. Everything else is cancelled, loopback included.--host-resolver-rules="MAP * ~NOTFOUND, EXCLUDE localhost[, EXCLUDE <dev host>]", because browser-internal fetches such as the spellcheck dictionary download bypass webRequest.ServerService.startServerrefuses), so browser and CLI clients cannot render tapes outside the block.isSessionTapeReplay. WorkspaceStore then applies tape events without side effects (skill refresh, gateway dialog, toasts) and skips the file-modifying-tool refresh.statfirst: 32 MiB recorder cap, regular files only.RangeErroron any line becomes a rejection.tests/e2e/fixtures/sessionTapes/perf-tape-replay.jsonlis synthetic, generated with the T2 builder byscripts/perf/generateTapeReplayFixture.ts. It contains probe image URLs (a.invalidhost and loopback127.0.0.1:47999) and a tool call with a non-builtin marker command.coderandghwith failing logged stubs, and wrapsgitwithGIT_ALLOW_PROTOCOL=file.Validation
make perf-tape-replaypasses locally and remotely.perf-summary.json→tapeReplay: 10 events, recorded duration 361 ms, first row ~0.8 s, last row ~1.3 s, 2 blocked probe requests (net::ERR_BLOCKED_BY_CLIENT). The send attempt shows the read-only refusal.strace -fcovered connect, sendto, sendmsg, execve and socket, with a capture server on 127.0.0.1:47999.originpointed at it.connectthat failed withENETUNREACHand sent no data.ghorcoderran, and the recorded tool command never ran.caught-up) show the refusal row. A stopped tape plays.ghegress. The harness isolation above fixed it.Risks
Low for normal use. Every new branch is gated on
XUM_REPLAY_TAPES, and onChat, send, model creation, telemetry and the API server behave as before when it is unset.Deferred (tracked in #5587)
Synchronous tape load on the main process.
fastpacing. Replaying real local tapes throughmake. Composer side effects of replayedrestore-to-input/auto-compaction-triggeredevents. Background network isolation outside the harness.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high• Cost:$29.69