Row-budget crush for updates/list_refhosts/openqa_overview

[plusky]

Summary

Caps the three unbounded MCP listings (updates, list_refhosts, openqa_overview) at a row budget (head 40 + tail 10 + first 50 middle anomalies, cap 100) so large queues/inventories no longer blow past [mcp] max_output_bytes. Adds --offset to updates and --limit/--offset to list_refhosts so any dropped middle slice stays recoverable.

Changes

• New crates/mtui-core/src/commands/row_budget.rs: deterministic crush (exact-dedup, head+tail+anomaly keep, order-preserving) plus row_notice naming the narrowing flags.
• updates: anomaly = status other than testing (unknown/missing kept, safe direction); new --offset, applied pre-crush (skip then truncate). list_refhosts: anomaly = non-free lock/pool claim; new --limit/--offset. openqa_overview: per-section crush, anomalies = non-passed/non-empty matches; --export still writes the full uncrushed overview.
• --json stays a valid array; over-cap output appends one trailing …[truncated N of M rows; …] line — strip lines starting with that prefix before parsing (documented in both --json helps, which flow into the MCP descriptions).
• Honesty note: row-cap is not byte-cap; huge rows can still hit the MCP byte backstop (covered by a test).

Checklist

• Commits follow Conventional Commits.
• cargo fmt --all --check is clean.
• cargo clippy --workspace --all-targets --all-features -- -D warnings is clean.
• cargo test --workspace passes.
• Feature matrix builds (--no-default-features, --all-features, compile-only).
• New/changed code covered (helper ~100% line-covered; crush/truncate/boundary/export-bypass paths asserted).
• User-visible changes recorded in CHANGELOG.md.
• Docs updated where relevant (docs/src/cli.md regenerated via cargo xtask gen-docs; --json helps document the wire protocol).

Related issues

Part 1 of 4 in the MCP token-reduction stack (row-budget crush for the 99k-token listing spikes). Merge order: log-fold → json-crush → reread-dedup → token-nudges. Expected trivial conflicts with siblings on CHANGELOG.md and crates/mtui-mcp/tests/it.rs; resolved at merge time in that order.

[codecov]

Codecov Report

❌ Patch coverage is 98.54148% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.83%. Comparing base (cba62ab) to head (bc1cdb2).

Files with missing lines	Patch %	Lines
crates/mtui-core/src/commands/openqa_overview.rs	91.82%	13 Missing ⚠️
crates/mtui-core/src/commands/list_refhosts.rs	99.66%	1 Missing ⚠️
crates/mtui-core/src/commands/row_budget.rs	99.70%	1 Missing ⚠️
crates/mtui-core/src/commands/updates.rs	99.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #610      +/-   ##
==========================================
+ Coverage   96.76%   96.83%   +0.07%     
==========================================
  Files         213      214       +1     
  Lines       61858    62943    +1085     
==========================================
+ Hits        59854    60951    +1097     
+ Misses       2004     1992      -12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[mimi1vx]

High priority blocker

• [crates/mtui-core/src/commands/updates.rs:369] --json emits invalid JSON when truncated. When output exceeds ROW_CAP (100 rows), appending the plain-text notice line …[truncated N of M rows; narrow with …] after the JSON array breaks standard JSON parsers (jq, serde_json, Python json.loads()). Any pipeline piping mtui updates --json | jq . or an MCP client parsing JSON tool results will fail with syntax/decode errors. Route the truncation notice to stderr/eprintln so stdout remains strictly valid JSON, or omit it when --json is active.
• [crates/mtui-core/src/commands/list_refhosts.rs:449] --json emits invalid JSON when truncated. Plain-text notice appended to stdout after the JSON array corrupts JSON parsing on list_refhosts --json. Route to stderr/eprintln or omit on --json.

Blocker

• [crates/mtui-core/src/commands/openqa_overview.rs:261] Redundant heap allocations during crush. single_incidents.clone() and group.versions.clone() duplicate large Vec<VersionResult> structures solely to extract deduplication keys. Pass &[VersionResult] or take ownership to avoid multiple bulk vector clones.
• [crates/mtui-core/src/commands/row_budget.rs:35] Optimize dedup key generation in updates. key_of currently runs serde_json::to_string(v) for every Value in the queue to insert into a HashSet<String>. For large queues, keying by a lightweight tuple (such as (id, status, priority)) avoids hundreds of JSON serialization allocations during dedup.

[mimi1vx]

Blockers

1. [P0] The row-crush truncation notice for --json mode goes to real eprintln!/stderr, which mtui-mcp never surfaces on a successful call. commands/updates.rs and commands/list_refhosts.rs write the notice to stderr, but McpSession::run_command only returns the captured display sink on success — stderr is not merged in. An MCP client requesting updates --json or list_refhosts --json on a large result silently gets an incomplete listing with no per-call signal that rows were dropped. This is inconsistent with the precedent already set by mtui-mcp/src/slim.rs's byte-cap (cap_output/truncation_notice), which accepts corrupting stdout's JSON validity specifically so the client is informed. Route the row-crush notice the same way (in-band, even if it costs JSON validity) before merging — this is the exact failure mode ("silent truncation") the row-budget feature exists to prevent.
2. [P1] commands/updates.rs::update_key mishandles an explicit "id": null row. field("id") returns Some("null") for an explicit null (not None, since the key is present), so such a row skips the rest-based exact-dedup fallback and is keyed only by (Some("null"), status, priority) — two genuinely different null-id rows sharing status/priority silently collapse into one. Check .is_null() alongside "key absent."
3. [P2] commands/row_budget.rs:94 has no compile-time or runtime guard on ROW_HEAD + ROW_TAIL <= ROW_CAP. anomaly_idx.truncate(ROW_CAP - ROW_HEAD - ROW_TAIL) will underflow (panic in debug, wrap to a huge length in release, silently defeating the cap) if a future edit changes any of the three constants without preserving the relation. Add const _: () = assert!(ROW_HEAD + ROW_TAIL <= ROW_CAP);.
4. [P2] No insta snapshot coverage for the new notice text or the crushed table/JSON shape — only ad-hoc contains/starts_with assertions. Per this project's testing conventions, display-output text contracts belong under insta so incidental wording drift is caught.

[mimi1vx]

BLOCKING: crates/mtui-core/src/commands/updates.rs:930-938 and list_refhosts.rs:148-155 — --json truncation is routed to eprintln!/stderr to keep stdout valid JSON, but MCP never captures real stderr (confirmed by this PR's own test). An MCP --json call over the row cap returns a silently truncated array with zero in-band signal, contradicting this project's own "never a silent cut" convention and regressing the guarantee the existing byte-cap mechanism (slim.rs::cap_output/truncation_notice) already provides.

BLOCKING: crates/mtui-core/src/commands/row_budget.rs:696-699 — anomaly retention is capped at 50 and kept in arbitrary positional order, not by severity. Real failures (failed jobs, locked hosts) can be dropped silently alongside routine rows with no distinguishing count in the notice.

BLOCKING: crates/mtui-core/src/commands/list_refhosts.rs:113-126 — probe_locks under --free still runs against every matched host before --offset/--limit windowing is applied, so paging doesn't reduce the SSH probe cost this feature exists to address.

FLAG: same root failure mode as #620/#611 — this PR silently drops information instead of signaling it in-band, mirroring #620's secret-redaction gap. Please resolve all three PRs under one shared pattern rather than as three separate patches to the same lesson.

[plusky]

Review-sweep responses (2026-09-12) — all findings addressed, each rework adversarially re-reviewed (GO) with red-proven tests, full workspace gates green. Cross-PR summary (per-PR diffs carry the detail):

• Fold repetitive fan-out output (run/reboot/perform) #608 log-fold: traceback-family keywords (traceback/stacktrace/keyerror/assertionerror), blank runs ≥3 fold, fanout_detail grouping-key doc+test.
• MCP token nudges: core profile, terser heavy-tool helps #609 token-nudges: get "full" restored, patch/write relpath examples, --json help made truthful for the pre-Row-budget crush for updates/list_refhosts/openqa_overview #610 tree (limit-capped array + limit hint, zero truncation claims); hunk yields to Row-budget crush for updates/list_refhosts/openqa_overview #610's exact text at land time.
• Row-budget crush for updates/list_refhosts/openqa_overview #610 json-crush: severity-ordered anomalies (u8, K/L kept counts in notice), window-before-probe in list_refhosts with pre-window totals in notice/footer.
• Collapse exact testreport_read re-reads to a notice #611 reread-dedup: additive deduped bool on read output (true on collapse, false on file text), documented; notice text unchanged.
• fix(mcp,testreport): stop steering transport unlock failures at --force #618 unlock-force: two-foreign-owners cancel test; ContendedSurface enum replaces paired check/scope strings, byte-identical rendering.
• feat(mcp): durable audit record of tool calls #620 audit-log+OTLP: real Debug negative pins, OTLP config_set body test, Full-via-dispatch test, always-fingerprint put/write/patch payloads as {bytes,sha256} (sha2, already in lock), resolve gated on auditing(), atomic 0600 create + tightening pre-existing sinks.
• fix(mcp): filter config_show on the headless surface #621 config-show: reuses pub sanitize_url (local dup deleted), edge-case parity tests, const subset assertions + url-defaults coverage test.

One systemic thread across #610/#611/#620: new I/O or truncation reuses the project primitives (spawn_blocking for blocking I/O, in-band truncation notices) — applied in each.

[mimi1vx]

Blocker (P0)

• crates/mtui-core/src/commands/list_refhosts.rs:473-486: --free windowing emits truncation notice on user-limited queries, breaking --json and fabricating anomaly totals. In list_refhosts, notice is triggered whenever (free && window_dropped > 0). When a caller explicitly requests --free --limit 10 --json, window_dropped is matched_total - 10 > 0, which forces a plain-text notice line …[truncated 140 of 150 rows (K/K anomalies kept); narrow with --limit/...] to be appended to stdout after the JSON array. This breaks JSON parsers for callers who explicitly narrowed their query. Furthermore, row_notice asserts that K/K anomalies kept were retained across all 150 hosts, even though the 140 unprobed hosts were never surveyed for locks. Only emit row_notice when the budget crush itself drops rows (crushed.truncated > 0), and decouple partial-inventory probe notes from row truncation.
• crates/mtui-core/src/commands/updates.rs:38-44: update_key does not handle JSON null IDs, falsely deduping distinct id-less rows. In update_key, let id = field("id"); produces Some("null") when "id": null because ToString formats Value::Null as "null". Consequently, id.is_none() evaluates to false, leaving rest as None. Any two distinct update rows that have "id": null and matching status and priority receive the exact same key (Some("null"), status, priority, None), causing the second row to be silently dropped as a duplicate. Filter out nulls: let id = v.get("id").filter(|v| !v.is_null()).map(ToString::to_string);.

Blocker (P1)

• crates/mtui-core/src/commands/row_budget.rs:59-67: row_notice reports middle-slice anomaly counts as whole-dataset counts. When total > ROW_CAP, scored is gathered strictly from (ROW_HEAD..tail_start). anomaly_total and anomaly_kept completely ignore anomalies in the head (first 40 rows) and tail (last 10 rows). If 5 failed jobs or locked hosts are located in the head and none in the middle slice, the notice reports …[truncated 50 of 150 rows (0/0 anomalies kept); narrow with …] despite 5 anomalies being clearly visible, misleading callers into believing no anomalies exist. Calculate anomaly_total and anomaly_kept across the complete dataset (0..total).
• crates/mtui-core/src/commands/updates.rs:397-404, list_refhosts.rs:482-486: Trailing non-JSON in-band notice corrupts stdout for standard JSON parsers and CLI pipelines. Appending …[truncated N of M rows …] directly to stdout breaks standard CLI JSON tooling (mtui updates --json | jq .) whenever output exceeds the row budget. Emit structured metadata / JSON envelope when --json is requested, rather than concatenating plain text to a serialized JSON array.

Blocker (P2)

• crates/mtui-core/src/commands/row_budget.rs:158-168: Omit (0/0 anomalies kept) when anomaly_total == 0. Formatting (0/0 anomalies kept) when no anomalies exist looks like an unhandled edge case or template defect. Omit the parenthetical when anomaly_total == 0.

[mimi1vx]

All four prior blockers (--json validity, stderr-routed truncation notice, --free window fabricating anomalies) are resolved and verified against PR head — I reverted each fix independently and confirmed the regression tests go red. Tests/clippy/fmt/doc-gen all clean. Approving.

[mimi1vx]

This PR is approved and was mergeStateStatus: CLEAN — but #621 (fix/410-config-show-surface) merged first via rebase and this branch now shows CONFLICTING against main. I don't have maintainer-edit access on this branch to rebase it myself.

@plusky could you rebase feat/mcp-json-crush onto current main and force-push? Once it's conflict-free I'll merge.

[mimi1vx]

and with rebase also squash commits ?