Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
4717136
feat(telemetry): record the retry prompt as gen_ai.input.messages
dhshah13 Sep 18, 2026
73d586b
feat(telemetry): record tool arguments on tool_call parts
dhshah13 Sep 18, 2026
19f3c5c
docs(tracing): describe input capture and tool arguments
dhshah13 Sep 18, 2026
4fb9cd7
docs(adr): annotate ADR 0050 and ADR 0108 for input capture and tool …
dhshah13 Sep 18, 2026
99f1399
fix(telemetry): charge and mark arguments lost to a redacted-away name
dhshah13 Sep 21, 2026
1c124ce
fix(runtime): hold accumulated tool input to maxToolInputSize
dhshah13 Sep 21, 2026
c697b84
fix(telemetry): replace runner env credential values in Level 3 content
dhshah13 Sep 21, 2026
a1c1851
fix(telemetry): repeat the runner env literal pass after the pipeline
dhshah13 Sep 21, 2026
7944734
docs(tracing): describe the runner env literal pass and shorten the A…
dhshah13 Sep 21, 2026
9f55a67
Merge main into feat/l3-input-and-arguments
dhshah13 Sep 23, 2026
67d08ab
docs(tracing): say fullsend.truncated reads by part type
dhshah13 Sep 23, 2026
38e3f39
Merge main into feat/l3-input-and-arguments
dhshah13 Sep 28, 2026
b800ee0
Merge branch 'main' into feat/l3-input-and-arguments
dhshah13 Oct 6, 2026
3ecc9b5
fix(telemetry): judge values nested under a secret-named member
dhshah13 Oct 6, 2026
1fd2b7d
fix(telemetry): one literal pass for Level 3 strings and execute_tool…
dhshah13 Oct 6, 2026
e7c1e44
fix(telemetry): judge keys under a secret-named member
dhshah13 Oct 6, 2026
f03024a
Merge main into feat/l3-input-and-arguments
dhshah13 Oct 6, 2026
558f060
fix(cli): mask a string holding a document with a secret-named member
dhshah13 Oct 6, 2026
60fa4db
fix(cli): judge a held document as exported, and fail closed
dhshah13 Oct 7, 2026
4d3c7ba
fix(cli): count stripped escape sequences as revealing; trim invisibles
dhshah13 Oct 7, 2026
96612ea
fix(cli): judge a held JSON string literal decoded too
dhshah13 Oct 7, 2026
d319570
fix(cli): mask whole document-shaped text that does not parse
dhshah13 Oct 7, 2026
69583ad
docs(tracing): a document-shaped string that does not parse is masked…
dhshah13 Oct 7, 2026
6ba0c63
fix(cli): bound the arguments walk before it starts
dhshah13 Oct 8, 2026
938a07c
fix(cli): judge tool arguments across their strings and folds
dhshah13 Oct 8, 2026
b1eb40b
fix(cli): judge across the strings whatever else drops the level
dhshah13 Oct 8, 2026
2b07a78
fix(cli): judge the strings joined by nothing as well
dhshah13 Oct 8, 2026
f713132
fix(cli): seek an exact value in the strings with white space aside
dhshah13 Oct 8, 2026
fa2f0b4
fix(cli): a key masked whole as a document names a secret too
dhshah13 Oct 8, 2026
bb50b21
refactor(cli): judge tool arguments per leaf and drop on two coarse r…
dhshah13 Oct 8, 2026
c8324df
refactor(cli): record only the listed members of a tool call's arguments
dhshah13 Oct 8, 2026
cbfe80e
fix(cli): scan listed numbers, reject trailing input, keep the record…
dhshah13 Oct 8, 2026
31f363d
Merge main into feat/l3-input-and-arguments
dhshah13 Oct 13, 2026
efa0465
fix(cli): scan a string the normalizer stripped as written, once for …
dhshah13 Oct 13, 2026
16c1e13
fix(cli): mask a string the normalizer stripped from whole, no count …
dhshah13 Oct 13, 2026
28055ba
test(cli): state the redaction guarantee as one invariant; mask what …
dhshah13 Oct 13, 2026
574e44f
fix(cli): treat the replacement character as a lost byte; a call not …
dhshah13 Oct 13, 2026
3c529ce
docs(guides): name the replacement character in the user guide
dhshah13 Oct 13, 2026
ed17071
docs(guides): a prompt cut for size alone is still recorded
dhshah13 Oct 13, 2026
967e57a
fix(cli): arguments listed by tool, URLs masked, policy drops unmarke…
dhshah13 Oct 14, 2026
d3c1d7e
fix(cli): mask a retry prompt cut inside a URL; escaped separators
dhshah13 Oct 14, 2026
69d849e
fix(cli): end a URL at a quote that closes a JSON string
dhshah13 Oct 14, 2026
845c0ee
fix(cli): mask a URL whole past its separator when a ? or # precedes …
dhshah13 Oct 15, 2026
36a8978
fix(cli): read a URL with an escaped colon or a percent-encoded separ…
dhshah13 Oct 15, 2026
3184209
fix(cli): read a URL separator with any part percent-encoded
dhshah13 Oct 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/ADRs/0050-distributed-tracing-instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,3 +178,16 @@ codex emit no call ids — becomes an `execute_tool` child of its iteration's
`agent` span, up to 1,024 per iteration, metadata only; the message record
on the `agent` span stays the content carrier. Sub-agent nesting (deferred item 1 above)
remains deferred.

**2026-09-18 — Level 3 input message and tool arguments ([ADR 0108](0108-tool-call-span-topology.md)):**
under the content gate, a retry iteration that carries validation feedback
records the prompt the runner composed as `gen_ai.input.messages`, and
`tool_call` parts on the message record carry the members of the call's
arguments that name what was called (a path, a pattern, a `Bash` command
whole, which can embed what it writes), not a `Write`, `Edit` or
`NotebookEdit` body, an `Agent` or `WebFetch` prompt or the input of an
MCP tool — Claude Code today
([#7414](https://github.com/fullsend-ai/fullsend/issues/7414)).
The rest of the model's input is not recorded
([reference](../guides/infrastructure/distributed-tracing.md#content-capture-level-3)).
The gate and the carrier are unchanged.
13 changes: 13 additions & 0 deletions docs/ADRs/0108-tool-call-span-topology.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,16 @@ convention bump.
call ids and results ([#7414](https://github.com/fullsend-ai/fullsend/issues/7414)).
- This settles the span-granularity question in #294; retention and access
remain open there.

## Annotations

**2026-09-18 — Tool arguments on the message record:** the Decision's "full
arguments next" is in place for the members that name what was called:
`tool_call` parts on the `agent` span's `gen_ai.output.messages` record
carry the call's path, pattern, command and the like, redacted, where the
runtime's stream provides them, Claude Code today, for its own tools (a
`Bash` command whole, which can embed what it writes); a `Write`, `Edit` or
`NotebookEdit` body, an `Agent` or `WebFetch` prompt and the input of an
MCP tool are not recorded. The list, the bound and the drop rules are in the
[reference](../guides/infrastructure/distributed-tracing.md#content-capture-level-3).
`execute_tool` spans still carry metadata only.
9 changes: 6 additions & 3 deletions docs/contributing/go-code.md
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,9 @@ function in `internal/cli/run.go` is the canonical implementation.
`_SECRET`, `_PASSWORD`, `_KEY`, `_CREDENTIALS`) with
`[REDACTED:<key>]`. Skip values shorter than
`minRedactableSecretLen` (currently 8) — short values like `"main"`
or `"true"` cause false-positive mangling.
or `"true"` cause false-positive mangling. `replaceEnvSecrets` is that
pass: it takes longer values first, so a value that contains another
is replaced whole.

2. **Scan `providerOnlyKeys` from the process environment
(`os.Getenv`) for credential literal values.** Provider-only
Expand All @@ -482,8 +484,9 @@ function in `internal/cli/run.go` is the canonical implementation.
can't reach them, which means the `RunnerEnv` scan in invariant 1
never sees them. Iterate `providerOnlyKeys`, read each value with
`os.Getenv`, and replace it the same way (skipping values shorter
than `minRedactableSecretLen`). A future credential class kept out
of `RunnerEnv` for the same reason needs the same treatment here.
than `minRedactableSecretLen`). `replaceEnvSecrets` does this in the
same longest-first pass as invariant 1. A future credential class kept
out of `RunnerEnv` for the same reason needs the same treatment here.

3. **Apply `security.SecretRedactor` as a fallback pass.**
The `RunnerEnv` and `providerOnlyKeys` scans only catch credentials
Expand Down
143 changes: 131 additions & 12 deletions docs/guides/dev/tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,11 +189,11 @@ near-zero-duration span marked `fullsend.tool.unmatched=true`. Events without
an id — pi and codex emit none — produce no span, so the child count can be
below `fullsend.tool_calls`, which counts every reported call, id or not; a
`server_tool_use` block on an `assistant` line produces no event at all (its
result never arrives as a `tool_result`), so it appears in neither count. The name passes through `security.OutputPipeline()`
— Unicode normalization, then secret redaction, the same pipeline as span
content — and is bounded to 256 bytes before it becomes the attribute; the
span name keeps at most 128 bytes of it. The call id is scanned through the
same pipeline and dropped from the span on any finding — never substituted,
result never arrives as a `tool_result`), so it appears in neither count. The name passes through `redactText`
— the runner env literal pass on both sides of `security.OutputPipeline()`,
the redaction span content gets — and is bounded to 256 bytes before it becomes the attribute; the
span name keeps at most 128 bytes of it. The call id goes through the
same `redactText` and is dropped from the span on any finding — never substituted,
since a masked id could collide with another call's — while the raw bounded
id still keys the open-call map, so correlation is unaffected.
The tracker records at most `maxToolSpansPerIteration` (1,024) spans per
Expand Down Expand Up @@ -225,22 +225,39 @@ renderer path.
The collector (`internal/cli/content_collector.go`) coalesces contiguous
text/reasoning deltas, maps tool use to `tool_call` parts and tool
results to `tool_call_response` parts (only the Claude parser emits
`ToolResultEvent` and call ids today — pi and codex emit neither,
`ToolResultEvent`, call ids and `ToolUseEvent.Arguments` today — pi and
codex emit none of them,
[#7414](https://github.com/fullsend-ai/fullsend/issues/7414); the schema's
required result field is `response`), redacts every
part through `security.OutputPipeline()` at assembly (redaction runs
part — `security.OutputPipeline()`, with `replaceEnvSecrets` for the values
of sensitive runner env keys and of `providerOnlyKeys`, in one pass, on both
sides of it (`redactText`; a string the normalizer stripped an escape
sequence or tag characters from is masked `***` whole — a colour code
ends at the next letter, so it can take a token's first letter, and a
title code's payload is text no pattern sees, so nothing cut from such a
string can be trusted, and no scan of it as written can stand in; so is a
string holding invalid UTF-8, since the span and attribute writers would
join the halves of a value such a byte splits, or the replacement
character a lossy decode leaves in its place, which splits a value the
same way; last, every URL's userinfo, query and fragment are masked,
`maskURLs`, on text the normalizer has folded and the patterns masked) —
at assembly (redaction runs
before the size budget — truncating first could split a secret past
recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the
final answer is what consumers judge) plus an 8 KiB per-tool-result
bound (tail-kept, redacted before the cut, the part marked
`fullsend.truncated`), with exact dropped-byte
accounting across content, tool names, summaries, responses, and part
ids, then holds the marshaled string to `maxEncodedContentBytes`
`fullsend.truncated`) and an 8 KiB per-call arguments bound (over it the
arguments are dropped whole and the part marked, since a cut object is not
JSON), with exact dropped-byte
accounting across content, tool names, summaries, responses, part
ids, and arguments dropped whole (charged as the kept encoding when the
bound drops them, otherwise as the redacted text they were scanned as; a
member the recorded list leaves out is not charged), then holds the marshaled string to `maxEncodedContentBytes`
(255,000 — just under the one size the pilot backend is proven to accept)
by trimming the oldest content again, measured on the encoding itself and
still charged in raw bytes (a separate, earlier boundary — the parser's 1 MiB
stream-line cap — skips oversized lines; an oversized `tool_result` line
still yields an empty part marked `fullsend.truncated`). None of the three
still yields an empty part marked `fullsend.truncated`). None of the four
content bounds is a measured backend limit; the constants' comments and
[Size limits](../infrastructure/distributed-tracing.md#content-capture-level-3)
name what blocks raising them. The collector emits
Expand All @@ -250,11 +267,113 @@ content and its marker attributes on the span before either
`finalizeAgentSpan` path can end it, so failed iterations keep their
content.

**Guarantee and limits.** For every string the record or an `execute_tool`
span exports, no 12-byte run of a known secret's body appears, read as raw
bytes, after the normalizer, or after decoding the record's JSON with every
string walked; a known secret is one a `SecretRedactor` pattern matches,
a runner env or provider-only env value the literal pass holds, or a
URL's userinfo, query or fragment.
`TestRedaction_NoSecretSurvivesAnyWrapping` (`internal/cli`) states it and
checks it over every surface (text, names, summaries, arguments on each
path, results, the input message, span names and ids, and a Claude call
with the summary the parser derives from it) and every wrapping a review
has constructed so far — terminal escapes before, around and inside a
secret, repeated and paired secrets, zero-width, bidi, tag, NUL and
variation-selector characters inside it, a URL's delimiters in
compatibility forms, separators escaped for the shell or JSON, a URL
percent-encoded whole or in part (its separator's colon or slashes, its
`@`, `?` or `#`), a
password quoted for the shell, a URL nested in another's path, an
invalid byte or the
replacement character inside it, an invalid byte after it with the
secret straddling the parser's summary cut, compatibility forms, and
noise, inside the record and across the feedback cut, in compatibility
forms or plain with 12 bytes before the cut.
Four limits are named there and here, and nowhere else: an env value
written in compatibility (fullwidth) forms, since the normalizer rewrites
it and the literal pass matches only what the env has; a secret split
across two parts or events; a secret that matches no pattern and is in no
env, among them a token in a URL's path, in a URL holding a double quote
that ends it (one that could close a JSON string, so a URL inside JSON
ends with its string), what follows the quote and userinfo whose `@`
follows it; a summary a pi or codex parser cut before the collector saw it, since
that cut can split a value the parser's own scan did not see joined (a
Claude summary is cut from a member the collector scans whole). A finding
inside the guarantee is a defect; one on this list is a documented limit.

`toolArguments` records the members of a call's input that
`recordedArguments` names for the tool — paths, patterns, commands, modes
and bounds; the reference lists them tool by tool — and nothing else: a
file body, an edit, a prompt, a notebook source or a todo list, any member
whose value is an object or an array, and the whole input of a tool the
list does not name (every `mcp__` tool) is dropped: scanned as text so a
secret in it still counts, then left out, neither charged nor marked,
since the list is the documented policy and the markers mean loss. A
`Bash` `command` is kept whole, so a heredoc or an inline script records
what it writes. A URL's userinfo, query and fragment are masked in every
recorded string (`maskURLs`, above), with a finding that costs a call its
summary. The guarantee is one sentence: the record holds these members of these tools,
each redacted as text, and nothing else of any call's input. The input is decoded
(one JSON object, white space at most after it) and each kept string, and
each kept number as its digits, goes through the text
pipeline on its own, since the redactor's patterns are written for plain
text and over serialised JSON miss an assignment that opens a string or
follows an escaped newline, and a value behind escaped quotes; the result
is encoded again, so key order, spacing and escapes are the encoder's. A
string the normalizer stripped from is masked `***` whole, as every
string is (above). Discarded content is redacted
first, here as everywhere: a dropped member is scanned as text — the
decoded string, or the compact encoding of an object or an array — so a
secret in a file body still counts, and costs the summary as a secret
found anywhere in the arguments does; so do arguments that could not be
judged at all — stripped, or holding invalid UTF-8 or the replacement
character — since the summary was cut from a member no scan saw whole.
Arguments that are not one JSON
object are scanned as text and dropped whole, charged; so are kept
members whose encoding exceeds `maxToolArgumentsBytes`, charged as
encoded. A call the stream reports without a name carries no arguments.
This happens once, when the event is handled; eviction and `Result` do
not rescan it. Arguments are kept only under a tool's exact name, plain
words no scan touches, so a name that redacts to nothing never had any.

The input message does not come from the stream. When `runAgent` composes
a retry prompt (`buildFeedbackPrompt`, under `feedback_mode: append`),
`attachInput` redacts it with the collector's pipeline — `redactFeedback`
ran before the feedback was sanitized and framed, and sanitizing can join
a token that scan saw split, so the recorded copy is pattern-scanned
again; a prompt the composition altered in a way that pass cannot see —
sanitized, or cut at `maxFeedbackBytes` while holding compatibility
characters or a URL `maskURLs` would mask, since the cut can take the `@`
that marks a password (`promptHidesAlterations`) — is recorded `***` whole; the
prompt the agent is sent is not changed, and the mask the first
scan leaves for a connection-string password of ten or more bytes matches
its own pattern and counts a finding —
and sets `gen_ai.input.messages` on the span
before the runtime starts, so each finalize path carries it. Its findings
join the iteration's, and its encoded size is charged against
`maxEncodedContentBytes`: the two content attributes of one span together
stay within the proven size.

**Consumer contract** (for eval scorers and other readers of
`run-telemetry.jsonl`): parse the `gen_ai.output.messages` attribute as
JSON; check `fullsend.content.truncated` / `fullsend.content.dropped_bytes`
before treating content as complete; masked secrets appear as the
redactor's mask tokens and are counted in `fullsend.content.redactions`.
redactor's mask tokens, or as `[REDACTED:<key>]` for a runner env or
provider-only value, and are counted in `fullsend.content.redactions`;
what was masked or rewritten for its shape — the normalizer's folds,
removals, strippings and invalid UTF-8, a retry prompt altered upstream,
a URL's userinfo, query or fragment — is counted apart in
`fullsend.content.sanitizations`.
A `tool_call` part's `arguments`, when present, is a JSON value (an object
for the tools seen so far); a `tool_call` part marked `fullsend.truncated`
had arguments that were dropped whole — on that part type the marker
never means a partial value, as it does on a tool result. `gen_ai.input.messages` is absent unless
the iteration is a retry that carried validation feedback — absence is
the normal case, not a gap. Masks `redactFeedback` left in the feedback
(`abcd...`, `***`, `[REDACTED:<ENV_KEY>]`) are in `gen_ai.input.messages`
too, but that function discards its findings, so they are not counted in
`fullsend.content.redactions`; the connection-string mask described above
is the known exception.
The attribute names and shapes above are the consumption contract — see the
[Tracing reference](../infrastructure/distributed-tracing.md#content-capture-level-3).

Expand Down
Loading
Loading