Skip to content

🤖 perf: document Report slowness and synthetic tape replay on the Profiling Xum page #5625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ThomasK33 merged 2 commits into from
Oct 4, 2026
Merged

🤖 perf: document Report slowness and synthetic tape replay on the Profiling Xum page #5625

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c09e444
fix(sessionTapes): stop claiming make perf-tape-replay sets the harne…
ThomasK33 Oct 4, 2026
638fbc5
docs(profiling): document Report slowness and synthetic tape replay
ThomasK33 Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/config/keybinds.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ This works when the **Performance flight recorder** experiment is on. It is also
| ------------------------------------------- | ------------------ |
| Save a local performance report (no upload) | `Ctrl+Alt+Shift+S` |

The report is a private folder under `~/.xum/perf/reports/` with the flight recorder snapshot, recent CPU profile captures and a `trace.json` you can open in [ui.perfetto.dev](https://ui.perfetto.dev). It contains no chat content, prompts, tool payloads or environment variables. The desktop app opens the folder in your file manager. `xum server` shows its path.
See [Report slowness](/reference/profiling#report-slowness) for what the report folder contains and what to review before you share it.

### Session tapes (experimental)

Expand Down
149 changes: 137 additions & 12 deletions docs/reference/profiling.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@ description: Record performance samples, CPU profiles and hang stacks to find ou

Xum has built-in tools to find out why the app or its backend is slow. All of them are opt-in, except hang stacks on the desktop app.

| Tool | Turn on | Output | Safe to share? |
| ------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| [Flight recorder](#flight-recorder) | Experiment **Performance flight recorder** | In-memory samples, read with `xum api perf get-flight-recorder-snapshot` | Yes, after review. It contains procedure names, script URLs and function names. |
| [Triggered CPU profiles](#triggered-cpu-profiles) | Same experiment | `.cpuprofile` and `.json` files in `~/.xum/perf/captures/` | Yes, after review. Profiles contain function names and script paths or URLs. |
| [Hang stacks](#hang-stacks-desktop) | Always on (desktop app) | `[diag]` lines in `~/.xum/logs/mux.log` | Yes, after review. |
| [Offline analyzer](#offline-analyzer) | Run it from a Xum source checkout | Markdown, JSON or folded stacks | Yes, after review. It shows function names and source locations from the profiles. |
| [Session tapes](#session-tapes) | Experiment **Session tapes** | JSONL files in `~/.xum/perf/tapes/` | No. Tapes contain your full chat. |
| Tool | Turn on | Output | Safe to share? |
| ------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| [Flight recorder](#flight-recorder) | Experiment **Performance flight recorder** | In-memory samples, read with `xum api perf get-flight-recorder-snapshot` | Yes, after review. It contains procedure names, script URLs and function names. |
| [Triggered CPU profiles](#triggered-cpu-profiles) | Same experiment | `.cpuprofile` and `.json` files in `~/.xum/perf/captures/` | Yes, after review. Profiles contain function names and script paths or URLs. |
| [Hang stacks](#hang-stacks-desktop) | Always on (desktop app) | `[diag]` lines in `~/.xum/logs/mux.log` | Yes, after review. |
| [Offline analyzer](#offline-analyzer) | Run it from a Xum source checkout | Markdown, JSON or folded stacks | Yes, after review. It shows function names and source locations from the profiles. |
| [Report slowness](#report-slowness) | Command **Report slowness**. Needs the flight recorder experiment | A folder in `~/.xum/perf/reports/` | Yes, after review. Profiles keep folder names below your home folder. |
| [Session tapes](#session-tapes) | Experiment **Session tapes** | JSONL files in `~/.xum/perf/tapes/` | No. Tapes contain your full chat. |

Paths on this page use `~/.xum`, the default Xum home. If you set `XUM_ROOT`, Xum uses that folder instead.

Expand Down Expand Up @@ -286,7 +287,7 @@ Exit status: 0 on success, 2 on usage errors, and 1 on any other failure, for ex

## Session tapes

A session tape records what one full chat subscription received from the backend, with its original timing. Tapes are for local performance replay. Xum records full chat subscriptions from every client of the backend, not only the Xum UI: ACP sessions and other API clients that load a whole chat also create tapes.
A session tape records what one full chat subscription received from the backend, with its original timing. Tapes are for your own local performance analysis. The [replay harness](#replay-a-synthetic-tape-contributors) replays only synthetic tapes. Xum records full chat subscriptions from every client of the backend, not only the Xum UI: ACP sessions and other API clients that load a whole chat also create tapes.

<Warning>
Tapes contain your full chat with no redaction: message and reasoning text, tool inputs and
Expand Down Expand Up @@ -355,6 +356,53 @@ Retention: Xum keeps the newest 20 tapes within 200 MiB. There is no age limit.

</Accordion>

### Replay a synthetic tape (contributors)

`make perf-tape-replay` replays a synthetic tape through the real desktop app and profiles how fast the chat renders. It is for Xum contributors and runs from a Xum source checkout. It replays only the committed synthetic fixture `tests/e2e/fixtures/sessionTapes/perf-tape-replay.jsonl`. The make target takes no tape path.

<Warning>
Never replay your own tapes from `~/.xum/perf/tapes/`. Never copy them into the repository, commit
them, or use them as evidence in issues or pull requests. Use only synthetic fixtures.
</Warning>

1. Run:

```bash
make perf-tape-replay
```

The target runs `make build`, then the Playwright Electron scenario `tests/e2e/scenarios/perf.tapeReplay.spec.ts` with one worker. It needs a display. On headless Linux, run `xvfb-run -a make perf-tape-replay`. Pass extra Playwright flags in `PLAYWRIGHT_ARGS`.

2. Find the results in `artifacts/perf/electron/tape-replay-<timestamp>/`:
- `chrome-cpu-profile.json`: a renderer CPU profile. The [offline analyzer](#offline-analyzer) can read it.
- `chrome-trace.json`: a Chrome trace.
- `react-profile.json`: React render timings.
- `perf-summary.json`: the run summary. Its `tapeReplay` object has the time to the first and last chat row (`firstRowMs`, `lastRowMs`), the tape's `eventCount` and `durationMs`, and `blockedRequests`.

To change the fixture, edit `tests/e2e/fixtures/sessionTapes/tapeReplayFixture.ts` and run `bun scripts/perf/generateTapeReplayFixture.ts`.

<Accordion title="How replay stays offline">

Xum serves a tape only inside the isolated test harness. The app replays only when both `XUM_E2E=1` and `XUM_REPLAY_HARNESS=1` are set. The Electron test fixture (`tests/e2e/electronTest.ts`) sets `XUM_E2E`, and the scenario sets `XUM_REPLAY_HARNESS` and `XUM_REPLAY_TAPES`. The Makefile sets none of them. Anywhere else, the chat shows the refusal `session tape replay runs only inside the perf harness`.

In replay mode, the desktop app does this:

- It blocks renderer network requests. Only `data:`, `blob:`, `devtools:` and local `file:` URLs load, so recorded image URLs are never fetched.
- It refuses model calls and external opens.
- It makes the chat read-only. A send shows a read-only error.

Replay mode does not take the app's other background network offline, for example git remote queries, `gh` and Coder CLI probes. The scenario does that:

- It removes variables that end in `_API_KEY`, `_AUTH_TOKEN` or `_BASE_URL` from the app's environment.
- It puts failing `gh` and `coder` stubs first on `PATH`.
- It wraps `git` so that git can use only the local file transport.

This is why Xum refuses replay outside the harness.

`XUM_REPLAY_TAPES` is a JSON map of workspace ID to tape path. The scenario sets it. It is not a user setting. Each tape path must be an absolute local path: a POSIX path that starts with `/`, or a Windows drive path such as `C:\`. Xum refuses UNC paths, `\\?\` prefixes, URLs and relative paths. This is a syntax check, not a folder restriction: `..` segments and symlinks are accepted. The file name must end in `.jsonl`, and the file must be a regular file.

</Accordion>

## Faster async context on Node 22 (`xum server`)

This section is for people who run `xum server` on Node 22.7 to 23. The Docker image already does this.
Expand Down Expand Up @@ -386,14 +434,91 @@ fi

To check that the flag works, [capture a backend profile by hand](#capture-a-profile-by-hand) while Xum is busy. The profile must have no frames from `node:internal/async_local_storage/async_hooks` and no `promiseInitHook` frames.

## Report slowness

**Report slowness** saves one local report folder with the data a developer needs to study a slowdown: the flight recorder snapshot, a trace you can open in Perfetto, recent CPU profiles and system details. Xum uploads nothing. You review the folder and decide whether to share it.

It needs the **Performance flight recorder** experiment ([turn it on](#turn-it-on)). Run it soon after the slowdown: the snapshot holds only the last 10 minutes. It works in the desktop app and in `xum server`.

### Save a report

1. Turn on the flight recorder.
2. Reproduce the slowdown, if you can.
3. Run **Report slowness** from the command palette (Help section), or press its shortcut. See [Keyboard shortcuts](/config/keybinds). From a terminal, you can run:

```bash
xum api perf-reports create
```

4. Xum writes the report folder and shows where it is:
- Desktop app: Xum opens the folder in your file manager and shows `Saved slowness report to <folder> and opened it in your file manager.`
- `xum server` in a browser: Xum shows `Saved slowness report to <folder>`.

In a chat view, the message is a toast with a copy button. It stays until you dismiss it. If the report fails, the toast title is `Report slowness failed`.

With the experiment off, the command does not appear and the shortcut does nothing. The shortcut also does nothing while a dialog is open. If you run the command again while Xum writes a report, Xum ignores it.

`xum api perf-reports create` takes no input. It returns `dir` (the folder), `revealed` (the desktop app opened it), `includedCaptures`, `skippedCaptures` and `totalBytes`. Errors:

- `PRECONDITION_FAILED` with "Report slowness needs the Performance flight recorder experiment": turn on the flight recorder.
- `CONFLICT` with "a slowness report is already being written": wait and try again.

### What the report contains

Xum writes the report to `~/.xum/perf/reports/<id>/`. An `id` looks like `20261003T013338383Z-1a2b3c4d`. The folder is private to your user (mode 0700, files 0600). Xum builds it in a hidden `.<id>.partial` folder and renames it when it is complete. Each report is at most 100 MiB.

| File | Content |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `snapshot.json` | The [flight recorder snapshot](#read-the-samples). Script URLs keep only scheme, host and path, and your home folder is written as `~`. |
| `trace.json` | The same timeline as Chrome trace-event JSON: event loop, slow oRPC calls, WebSocket waits, renderer long frames and slow input, trips and captures. Open it in [ui.perfetto.dev](https://ui.perfetto.dev) or the Chrome DevTools Performance panel. |
| `environment.json` | `version` (format version), `xumVersion`, `gitCommit`, `mode` (`desktop` or `server`), `platform`, `arch`, `osRelease`, `versions` (`node`, `v8`, plus `electron` and `chrome` in the desktop app), `enabledExperiments`, `recorderStatus` and `createdAtMs`. |
| `README.txt` | What each file holds, and what the report leaves out. |
| `captures/` | Up to the 10 newest [CPU profile captures](#find-captures) (`<id>.cpuprofile` and `<id>.json`) that fit in the size limit. `captures/manifest.json` lists the included captures, and the left-out captures with a reason. |
| `hangs.json` | Desktop app only. Up to 10 recent [hangs](#hang-stacks-desktop): time, duration and the function names of the stack. No URLs, file paths or line numbers. |
| `app-metrics.json` | Desktop app only. CPU and memory for each Electron process, from `app.getAppMetrics()`. |

The report never contains session tapes, chat content, prompts, tool payloads, session history or environment variables.

Xum does not delete old reports. Delete folders in `~/.xum/perf/reports/` yourself.

<Accordion title="Why a capture was left out">

`captures/manifest.json` gives one `reason` for each left-out capture:

| `reason` | Cause |
| -------------------- | -------------------------------------------------------------------------------------------- |
| `count-cap` | The capture is older than the 10 newest captures. |
| `size-cap` | The capture did not fit in 100 MiB. Xum then leaves out every older capture too. |
| `missing` | The profile file is gone. |
| `unreadable` | Xum could not read or copy the profile. |
| `not-a-regular-file` | The profile is a symlink, has more than one hard link, or is not a regular file. |
| `unscrubbable` | Xum could not safely rewrite the URLs and home folder in the profile, so it did not copy it. |

A capture without a profile (a skipped capture) is copied as its `.json` metadata only.

</Accordion>

### Review before you share

<Warning>
Path redaction in the report is partial. It is not anonymization. In copied CPU profiles, script
URLs lose their query, fragment and credentials, and your home folder is written as `~`. Folder
names below your home folder stay, for example `~/<project folder>/...`. The toast and the
`xum api` result show the full absolute path of the report folder. Review every file before you
share the report.
</Warning>

The report also contains what every snapshot contains (see [Privacy and cost](#privacy-and-cost)): oRPC procedure names and error codes, script URLs, function names, invoker strings, event names and element tag names. `environment.json` adds the enabled experiments and system versions.

## Report a performance problem

Collect these and attach them to the issue:

1. Numbers from the flight recorder. Paste excerpts of the snapshot, for example `jq '.trips'` and the last samples, not the full file.
2. Analyzer output for your captures: the leaderboard, or a diff against a good run.
3. The `.cpuprofile` files and their `.json` metadata from `~/.xum/perf/captures/`.
4. The `[diag] renderer unresponsive` lines from `~/.xum/logs/mux.log`, for hangs in the desktop app.
1. A [Report slowness](#report-slowness) folder. Review it first.
2. Numbers from the flight recorder. Paste excerpts of the snapshot, for example `jq '.trips'` and the last samples, not the full file.
3. Analyzer output for your captures: the leaderboard, or a diff against a good run.
4. Without a report: the `.cpuprofile` files and their `.json` metadata from `~/.xum/perf/captures/`.
5. The `[diag] renderer unresponsive` lines from `~/.xum/logs/mux.log`, for hangs in the desktop app.

Review everything before you share it. Profiles contain function names and script paths or URLs, as all V8 profiles do. Snapshots contain procedure names, script URLs and function names. Log lines contain the stack and page URL.

Expand Down
Loading
Loading