Skip to content

Add capture workflows, one-turn /trace, and always-on AGENTPROF_AUTO_TRACE recording - #18

Merged
dreveman merged 4 commits into
mainfrom
dev/dreveman/record-workflows
Nov 4, 2026
Merged

dreveman merged 4 commits into
mainfrom
dev/dreveman/record-workflows

Conversation

@dreveman

@dreveman dreveman commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Summary

This stack makes it possible to get an AgentProf trace in the three situations
users actually hit, without changing how existing manual recording works:

  1. No trace yet — a new capture_an_agent_run AI workflow walks an agent
    (or user) through installing the right recording plugin, capturing one
    task, and verifying the artifact before analysis.
  2. Just one prompt — /trace <prompt> in Pi and Claude Code records a
    single turn and prints the saved trace path.
  3. Nobody at the keyboard — AGENTPROF_AUTO_TRACE=1 +
    AGENTPROF_TRACING_DIR=<folder> makes every harness session a script
    launches record start-to-finish into one attributable folder.

Changes

AI workflow: capture a run

  • New capture_an_agent_run.md in the Perfetto skill overlay: consent and
    content-policy gates first, per-harness install + record + verify steps
    (Pi, Claude Code, Codex, Muse), then handoff to investigate_agent_run
    via inventory.sql. Routed in the skill router (patch 0009;
    0001–0008 only renumber).
  • investigate_agent_run now points here when there is no recording yet.
  • Fix: .gitignore's *.agentprof silently ignored new files under
    ai/skills/dev.agentprof/; added an exemption so future workflows show up
    in git status.

/trace <prompt> (Pi, Claude Code)

  • Pi: starts the capture, submits the prompt with sendUserMessage, and
    publishes from agent_settled so child sessions are included, then prints
    the trace path. Refuses while already recording or busy.
  • Claude: the mod intercepts the marker at turn.start (starting before the
    prompt is recorded, so the turn isn't marked partial), stores the prompt
    without the marker, and stops at turn.complete; the pending turn ID is
    checkpointed so a mod reload mid-turn still stops the recording. If a
    recording is already active the turn joins it.
  • Codex and Muse have no host mechanism to run a submitted prompt under
    plugin control, so they keep tracing start/tracing stop (documented).

Always-on scripted capture (all four harnesses)

  • AGENTPROF_AUTO_TRACE=1 records every top-level session that inherits the
    environment; =0 suppresses automatic capture for one launch and wins over
    the legacy switches (PI_TRACING, AGENTPROF_CODEX_AUTO_START,
    AGENTPROF_MUSE_AUTO_START, Claude auto_start).
  • AGENTPROF_TRACING_DIR sets the default output folder for all recordings
    without an explicit path. Automatic files are named
    <harness>-<full-session-id>-<timestamp>-<unique>.pftrace; child sessions
    keep joining their parent's trace.
  • Codex and Muse runtime bundles are regenerated.

Docs

  • All four harness READMEs, the root README, and the capture workflow
    (including a "scripted or always-on capture" section recommending pairing
    with AGENTPROF_CAPTURE_CONTENTS=0 for scripted fleets).

Behavior notes

  • Claude: the model receives the /trace prompt text as typed (the mod
    cannot rewrite a submitted prompt); only the stored prompt omits the marker.
  • Pi trace filenames from startup recording now contain the full session ID;
    an order-dependent test assertion was fixed to pair capture/flow events
    within a single trace file.

Tests

  • New tests per harness: Pi /trace driver test (one submission, published
    at settle, refusal while recording), Pi AGENTPROF_AUTO_TRACE config +
    end-to-end tests, two Claude /trace hook tests, Claude/Codex/Muse
    auto-trace tests including =0 suppression and tracing-dir relocation.
  • Full Pi, Claude, Codex, and Muse suites pass with a locally built Trace
    Processor; plugin bundle --check (Codex, Muse, Claude) passes; skills
    overlay and SQL tests pass; license check and git diff --check clean
    (except two pre-existing trailing-whitespace lines inside the generated
    patch context).

…ing a trace

The existing AgentProf workflows all assume a recording already exists and
explicitly stop before launching or recording a harness, leaving users to
find install and capture instructions on their own. Add a front-door
workflow that covers the full path from no trace to a verified artifact:
confirm the harness, task, content policy and output path before changing
anything; install the recording integration for Pi, Claude Code, Codex or
Muse with its version and trust prerequisites; record one task with
per-harness controls and measurement expectations; then verify the
published `.pftrace` with `inventory.sql` before handing off to
`investigate_agent_run`.

Route the new workflow from the Perfetto skill router (patch `0009`;
patches `0001`-`0008` only renumber) and point `investigate_agent_run` at
it when there is no recording yet. Exempt `ai/skills/dev.agentprof/` from
the `*.agentprof` ignore rule, which silently hid new workflow files from
`git status`, and extend `test_skills_overlay` to guard the per-harness
install steps, the consent-before-install ordering, and the verification
handoff.

Validation: `tools/test_skills_overlay.py` (11 tests); `tools/test_skills_sql.py`
with a locally built Trace Processor over all four reviewed harness
recordings (15 tests); `tools/release` unit tests (22 tests); license,
generated SQL export and icon checks; `git diff --check`.
Recording one prompt previously meant manually starting a capture, submitting
the task, and stopping afterward. `/trace <prompt>` now starts a recording,
runs just that prompt, stops when the turn finishes, and prints the trace
path, using the harness's default output path.

In Pi the command starts the capture, submits the prompt with
`sendUserMessage`, and publishes from `agent_settled` so child sessions are
included. In Claude Code the mod intercepts the marker at `turn.start`
(starting before the prompt is recorded, so the turn is not marked partial),
stores the prompt without the marker, and stops at `turn.complete`; the
pending turn id is checkpointed so a mod reload mid-turn still stops the
recording. Both refuse to take over an active recording (Pi errors, Claude
lets the turn join the existing capture). Codex and Muse have no host
mechanism that can run a submitted prompt under plugin control, so they keep
using `tracing start`/`tracing stop` around the turn; the capture workflow
and both harness guides document all of this.

Validation: Pi `/trace` driver test (submits one stripped prompt, publishes
one trace at settle, refuses while recording) and two Claude hook tests
(one-turn record with saved-path log; active-recording and bare-marker
cases); full Pi, Claude, Codex and Muse package suites (123 tests) with a
locally built Trace Processor; skills overlay and SQL tests; license check;
`git diff --check`.
…apture

Scripts and launchers that spawn harness sessions cannot type recording
controls, and each harness had a different partial answer (`PI_TRACING`,
`AGENTPROF_CODEX_AUTO_START`, `AGENTPROF_MUSE_AUTO_START`, a Claude plugin
option). Two cross-harness variables now cover the fleet case:
`AGENTPROF_AUTO_TRACE=1` records every top-level session from start to end
in any harness that inherits the environment (`=0` suppresses automatic
capture for one launch, winning over the legacy switches), and
`AGENTPROF_TRACING_DIR` sets the default output folder for all recordings
without an explicit path. Automatic files are named
`<harness>-<full-session-id>-<timestamp>-<unique>.pftrace` so a folder of
scripted traces is attributable without opening them; child sessions keep
joining their parent's trace.

Pi parses both variables in `applyEnvOverrides` (auto wins over
`PI_TRACING` in both directions), resolves the tracing dir as the recorder
output folder, and names startup recordings with the full session ID.
Claude starts from `session.start` when the variable is set and relocates
default writer paths. Codex threads `auto_trace`/`tracing_dir` through the
hook into the collector and `Journal`. Muse honors the switch at
`SessionStart` and inserts the folder into `control`'s output resolution.
Regenerate the Codex and Muse runtime bundles, document the pair in all
four harness guides, the root README, and the capture workflow, and pair
the order-dependent Pi capture/flow assertions within a single trace file
now that auto-start filenames changed capture sort order.

Validation: new Pi config and end-to-end auto-trace tests, Claude
session-start/manual-relocation tests, Codex and Muse auto-trace hook
tests (including `=0` suppression); full Pi, Claude, Codex and Muse suites
(230+ tests) with Node 24 and a locally built Trace Processor; plugin
bundle `--check` for Codex, Muse and Claude; skills overlay and SQL
tests; license check; `git diff --check`.
Regenerate the integrity digest in `marketplace.json` with
`node tools/check-muse-marketplace.mjs --update` after the Muse package
changes in the auto-trace commit (`record.ts`, tests, README, and the
rebuilt runtime bundle). Every byte of `packages/muse-tracing/` is
covered by this digest, so any package change requires it; without this,
CI's `npm run check` fails in the traces job.

Validation: `node tools/check-muse-marketplace.mjs` passes, and the full
`npm run check` component sequence (licenses, icon, skills SQL, all
typechecks, and plugin bundle `--check`s) passes locally.
@dreveman
dreveman merged commit 9610be7 into main Nov 4, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant