Repository navigation
Add capture workflows, one-turn /trace, and always-on AGENTPROF_AUTO_TRACE recording - #18
Merged
Merged
Conversation
…ing a trace The existing AgentProf workflows all assume a recording already exists and explicitly stop before launching or recording a harness, leaving users to find install and capture instructions on their own. Add a front-door workflow that covers the full path from no trace to a verified artifact: confirm the harness, task, content policy and output path before changing anything; install the recording integration for Pi, Claude Code, Codex or Muse with its version and trust prerequisites; record one task with per-harness controls and measurement expectations; then verify the published `.pftrace` with `inventory.sql` before handing off to `investigate_agent_run`. Route the new workflow from the Perfetto skill router (patch `0009`; patches `0001`-`0008` only renumber) and point `investigate_agent_run` at it when there is no recording yet. Exempt `ai/skills/dev.agentprof/` from the `*.agentprof` ignore rule, which silently hid new workflow files from `git status`, and extend `test_skills_overlay` to guard the per-harness install steps, the consent-before-install ordering, and the verification handoff. Validation: `tools/test_skills_overlay.py` (11 tests); `tools/test_skills_sql.py` with a locally built Trace Processor over all four reviewed harness recordings (15 tests); `tools/release` unit tests (22 tests); license, generated SQL export and icon checks; `git diff --check`.
Recording one prompt previously meant manually starting a capture, submitting the task, and stopping afterward. `/trace <prompt>` now starts a recording, runs just that prompt, stops when the turn finishes, and prints the trace path, using the harness's default output path. In Pi the command starts the capture, submits the prompt with `sendUserMessage`, and publishes from `agent_settled` so child sessions are included. In Claude Code the mod intercepts the marker at `turn.start` (starting before the prompt is recorded, so the turn is not marked partial), stores the prompt without the marker, and stops at `turn.complete`; the pending turn id is checkpointed so a mod reload mid-turn still stops the recording. Both refuse to take over an active recording (Pi errors, Claude lets the turn join the existing capture). Codex and Muse have no host mechanism that can run a submitted prompt under plugin control, so they keep using `tracing start`/`tracing stop` around the turn; the capture workflow and both harness guides document all of this. Validation: Pi `/trace` driver test (submits one stripped prompt, publishes one trace at settle, refuses while recording) and two Claude hook tests (one-turn record with saved-path log; active-recording and bare-marker cases); full Pi, Claude, Codex and Muse package suites (123 tests) with a locally built Trace Processor; skills overlay and SQL tests; license check; `git diff --check`.
…apture Scripts and launchers that spawn harness sessions cannot type recording controls, and each harness had a different partial answer (`PI_TRACING`, `AGENTPROF_CODEX_AUTO_START`, `AGENTPROF_MUSE_AUTO_START`, a Claude plugin option). Two cross-harness variables now cover the fleet case: `AGENTPROF_AUTO_TRACE=1` records every top-level session from start to end in any harness that inherits the environment (`=0` suppresses automatic capture for one launch, winning over the legacy switches), and `AGENTPROF_TRACING_DIR` sets the default output folder for all recordings without an explicit path. Automatic files are named `<harness>-<full-session-id>-<timestamp>-<unique>.pftrace` so a folder of scripted traces is attributable without opening them; child sessions keep joining their parent's trace. Pi parses both variables in `applyEnvOverrides` (auto wins over `PI_TRACING` in both directions), resolves the tracing dir as the recorder output folder, and names startup recordings with the full session ID. Claude starts from `session.start` when the variable is set and relocates default writer paths. Codex threads `auto_trace`/`tracing_dir` through the hook into the collector and `Journal`. Muse honors the switch at `SessionStart` and inserts the folder into `control`'s output resolution. Regenerate the Codex and Muse runtime bundles, document the pair in all four harness guides, the root README, and the capture workflow, and pair the order-dependent Pi capture/flow assertions within a single trace file now that auto-start filenames changed capture sort order. Validation: new Pi config and end-to-end auto-trace tests, Claude session-start/manual-relocation tests, Codex and Muse auto-trace hook tests (including `=0` suppression); full Pi, Claude, Codex and Muse suites (230+ tests) with Node 24 and a locally built Trace Processor; plugin bundle `--check` for Codex, Muse and Claude; skills overlay and SQL tests; license check; `git diff --check`.
Regenerate the integrity digest in `marketplace.json` with `node tools/check-muse-marketplace.mjs --update` after the Muse package changes in the auto-trace commit (`record.ts`, tests, README, and the rebuilt runtime bundle). Every byte of `packages/muse-tracing/` is covered by this digest, so any package change requires it; without this, CI's `npm run check` fails in the traces job. Validation: `node tools/check-muse-marketplace.mjs` passes, and the full `npm run check` component sequence (licenses, icon, skills SQL, all typechecks, and plugin bundle `--check`s) passes locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This stack makes it possible to get an AgentProf trace in the three situations
users actually hit, without changing how existing manual recording works:
capture_an_agent_runAI workflow walks an agent(or user) through installing the right recording plugin, capturing one
task, and verifying the artifact before analysis.
/trace <prompt>in Pi and Claude Code records asingle turn and prints the saved trace path.
AGENTPROF_AUTO_TRACE=1+AGENTPROF_TRACING_DIR=<folder>makes every harness session a scriptlaunches record start-to-finish into one attributable folder.
Changes
AI workflow: capture a run
capture_an_agent_run.mdin the Perfetto skill overlay: consent andcontent-policy gates first, per-harness install + record + verify steps
(Pi, Claude Code, Codex, Muse), then handoff to
investigate_agent_runvia
inventory.sql. Routed in the skill router (patch0009;0001–0008only renumber).investigate_agent_runnow points here when there is no recording yet..gitignore's*.agentprofsilently ignored new files underai/skills/dev.agentprof/; added an exemption so future workflows show upin
git status./trace <prompt>(Pi, Claude Code)sendUserMessage, andpublishes from
agent_settledso child sessions are included, then printsthe trace path. Refuses while already recording or busy.
turn.start(starting before theprompt is recorded, so the turn isn't marked partial), stores the prompt
without the marker, and stops at
turn.complete; the pending turn ID ischeckpointed so a mod reload mid-turn still stops the recording. If a
recording is already active the turn joins it.
plugin control, so they keep
tracing start/tracing stop(documented).Always-on scripted capture (all four harnesses)
AGENTPROF_AUTO_TRACE=1records every top-level session that inherits theenvironment;
=0suppresses automatic capture for one launch and wins overthe legacy switches (
PI_TRACING,AGENTPROF_CODEX_AUTO_START,AGENTPROF_MUSE_AUTO_START, Claudeauto_start).AGENTPROF_TRACING_DIRsets the default output folder for all recordingswithout an explicit path. Automatic files are named
<harness>-<full-session-id>-<timestamp>-<unique>.pftrace; child sessionskeep joining their parent's trace.
Docs
(including a "scripted or always-on capture" section recommending pairing
with
AGENTPROF_CAPTURE_CONTENTS=0for scripted fleets).Behavior notes
/traceprompt text as typed (the modcannot rewrite a submitted prompt); only the stored prompt omits the marker.
an order-dependent test assertion was fixed to pair capture/flow events
within a single trace file.
Tests
/tracedriver test (one submission, publishedat settle, refusal while recording), Pi
AGENTPROF_AUTO_TRACEconfig +end-to-end tests, two Claude
/tracehook tests, Claude/Codex/Museauto-trace tests including
=0suppression and tracing-dir relocation.Processor; plugin bundle
--check(Codex, Muse, Claude) passes; skillsoverlay and SQL tests pass; license check and
git diff --checkclean(except two pre-existing trailing-whitespace lines inside the generated
patch context).