feat(reporting): add quantitative skill performance dashboard

[chrisknvidia]

Summary

Closes #170.

Saved reports have to be inspected one at a time to compare agent skill quality and resource usage across models and harnesses. This PR adds an optional Python/Streamlit dashboard, skillevaluator dashboard, that loads canonical Tier 3 JSON, consolidated reports, local catalogs, or retained Tier 3 run directories. It shows:

• Performance: one sortable row per run, agent (harness and model), recorded feature configuration, and condition. Each row has score, pass@k, tokens, agent execution time, and USD cost as totals and as per-trial means.
• With vs without skill: paired with-minus-without differences (absolute and percent) for score, pass@k, tokens, time, and cost. A Comparison reason column says why a pair has no differences.
• Filters and exports: filters for skill, model, harness, harness features, condition, and status. Both tables download as CSV.

Measurement rules:

• Missing measurements stay blank and are never treated as zero.
• Conditions pair only within their originating report and agent occurrence, and only with the same feature configuration, dataset digest, attempt policy, and matching successful trial coverage.
• Repeated attempts pair by case, as main's HTML report does: each attempt needs its own trial ID.
• Agent time comes from canonical agent_seconds (fix: target dependency coverage and Tier 3 result integrity #196) or from retained agent_execution, under the same timezone-aware rule main uses. Cost appears only when a retained run records agent_result.cost_usd.
• Harness features and adverse conditions are import-only metadata. No evaluator command writes them today, and the dashboard tells the user so.

Loading and safety:

• Loading is bounded. External or symlinked retained artifacts are rejected, and one bad input does not hide the others.
• Inputs are read once per server and reused while you filter. Reload reports re-reads them from disk.
• Browser uploads accept 16 MB (16,000,000 bytes); local JSON reads accept 16 MiB.
• The launcher binds to 127.0.0.1 and starts Streamlit from its own package directory with python -P. It disables usage telemetry and the first-run email prompt, uses the next free port unless --port is given, and forwards SIGTERM/SIGHUP to Streamlit.

The dashboard reads saved outputs only. It never starts evaluations or estimates prices.

Changes

Area	What changed
CLI	skillevaluator dashboard [PATHS...] [--port N] [--no-browser] under Core workflows; dashboard optional extra (streamlit>=1.61,<2; 1.61 is the first release that honours the allowed-hosts setting)
Data	reporting/dashboard_data.py: normalization, pairing, fail-closed totals, bounded loaders, row dedupe
UI	reporting/dashboard_app.py (cached loading, filters, tables, CSV) and reporting/dashboard_ui/skillevaluator_dashboard.py (isolated Streamlit entry script)
Launcher	reporting/dashboard.py
Docs	Performance dashboard section in docs/reports.mdx; docs/cli-reference.mdx; dashboard row in the docs/installation.mdx extras table; CHANGELOG; third-party notices

Follow-up commits on this branch

After the original five commits, the branch was merged with main (latest d891b69) and got 18 follow-up commits from an enhancement pass, then 13 more that address a 14-finding inline review (all threads resolved), a fix for #77's compacted pruned-agent conditions, and a CI-only test fix. Their details are in the PR comments and review threads.

Verification

• Signed-off commits; documentation, changelog and third-party notices updated.

• Final head 2c0c013 (main d891b69 merged): all 17 CI checks pass, including Tests (Python 3.12 and 3.13), Tier 2 on Windows and macOS, DCO, Gitleaks and CodeQL. Earlier, at 4a945f8, two launcher tests failed only in the serial Python 3.12 coverage job, because they stubbed find_spec for all modules; 92e6219 stubs only the Streamlit lookup. Local full suite (Python 3.13, Streamlit 1.65) at 4a945f8: 13,090 passed, 22 skipped. There were 6 local failures, all outside the dashboard: 2 test_harbor_secure_docker_environment sidecar tests that fail the same way on main in this environment, and 4 bridge/preflight timing tests that pass on rerun. After the last merge, 5,554 reporting/CLI/evaluation/Tier 3 tests passed.

• Dashboard and CLI gate tests/reporting tests/test_cli.py tests/golden tests/test_oss_packaging.py: passed. Includes producer-contract tests that run main's canonical exporter and check that the dashboard's totals and deltas match the report's resources for 1 and 3 attempts.

• ruff check src tests, ruff format on the PR files, uv lock --check, git diff --check, and fern check (no errors; the authenticated redirect check is skipped without a Fern token).

• Launcher behaviour checked live against Streamlit 1.65:

  • a streamlit.py in the current directory no longer runs;
  • a project .streamlit/config.toml can't open CORS;
  • no email prompt on first run;
  • the default launch moves to 8502 when 8501 is busy;
  • SIGTERM to the launcher stops the server.

• Original author verification at 7b0448f:

  • 210 UI matrix cases;
  • 33 browser journeys across Chromium, Firefox and WebKit;
  • 24 CLI/runtime cases;
  • a retained real dual-arm evaluation checked against raw-artifact arithmetic;
  • CI green on Python 3.12 and 3.13.

  These were not all re-run after the follow-up commits; the automated suites above were.

• No credentials, private datasets or proprietary benchmark content added.

No provider-backed agent evaluation or billing request ran. Physical mobile/Safari devices, assistive technology, and AppTest upload support on the Streamlit 1.61 floor were not tested; the upload tests skip if AppTest lacks file_uploader.

Deferred follow-ups

• Adverse-vs-without comparisons, once an evaluator produces adverse arms.
• Evaluator commands writing per-trial cost and dashboard_metadata.harness_features, which would make those columns populate without hand edits.
• Aggregating comparable deltas across runs, and charts. [FEA]: create a tables and rows quantitative dashboard to compare agent skill performance #170 asks for tables.
• Whether the browser upload limit (16 MB) and the local file limit (16 MiB) should be one value.

Outstanding review gate

The original author asked for dedicated Bugbot and Security Review passes, which were unavailable in their session. A multi-area review with adversarial verification has since run on this branch (see the review threads), but it does not replace those specific passes if they are still required.

Release Impact

• Updated CHANGELOG.md

🤖 Generated with Claude Code

[rng1995]

Follow-up enhancements and fixes (pushed to this branch)

@chrisknvidia: I merged main into the branch, then added 18 follow-up commits from a review and enhancement pass. Every commit is signed off. The full local suite passes (11,861 tests); the only 2 failures are sidecar tests that fail the same way on main in my environment. Head is caeffe0. Each item below is its own commit, so any of them can be reverted on its own.

Merges

• 3d7b80d and ef1a827 merge main (fix: target dependency coverage and Tier 3 result integrity #196, then a53af93 with feat(tier3): run hardened Docker trials under Kata Containers #182, fix(tier3): derive accuracy scores from criterion verdicts #190 and fix(tier3): exclude failed skill reads from execution credit #201). Only CHANGELOG.md conflicted, and both sides were kept.

Correctness

• f31c9af: read canonical per-trial agent_seconds (added in fix: target dependency coverage and Tier 3 result integrity #196) as agent time.
  • Canonical trials never contain duration_seconds or cost_usd, so agent time and its delta were blank for every uploaded report.
  • The retained-run path now uses main's agent_execution_seconds, which treats timestamps without a timezone as unavailable, as the HTML report does.
  • A new producer-contract test runs main's exporter and checks that the dashboard's totals and deltas equal the report's resources.
• a6084f0: pair repeated Harbor attempts by case.
  • With --n-attempts N, Harbor names trials <task>__<shortuuid>, so names carry no attempt number. _trial_keys therefore returned None, and every score, pass@k and comparison went blank.
  • Each attempt with a distinct trial ID now counts as recorded identity. This is the same "matching case counts" rule main's HTML report uses.
  • Still fail-closed: missing or empty trial IDs, a trial ID shared across cases, mixed explicit and implicit attempts, and differing explicit attempt sets.
  • Trial-name parsing (…attempt1) is gone, because it misread case IDs such as retry-attempt1.
  • This reverses your "require recorded identity" rule, so please check it.
• dfa7288: a trajectory over the report loader's 2 MiB cap no longer blocks score comparisons.
  • Main keeps the score and only drops that trial's tokens.
  • Every other artifact_loading reason still blocks.
• d3f1029: a count of 2**64 or more crashed Arrow conversion and blanked the whole page. Counts are now capped at 1,000,000 and go through the invalid-coverage path. An unreadable trials/ directory now fails only its own row instead of dropping the whole run.

Performance

• 4a15034: cache inputs across Streamlit reruns.
  • Measured with st.cache_data on a 1,024-trial retained tree plus 300 reports: a filter click went from 2.6 s to 69 ms.
  • Uploads are keyed by file_id.
  • Download buttons use on_click="ignore", so a download no longer reruns the app.
  • A Reload reports button re-reads inputs from disk.
  • Adds upload-path tests, including the exact 16,000,000 / 16,000,001-byte boundary.
• 7689b38: when inputs overlap (a directory plus a file or run inside it, or a latest symlink), each row is loaded once. Before, every pair turned into "ambiguous duplicate conditions". Byte-identical uploads are skipped with a notice.

Safety

• 69c5bd7: report notices are shown as plain text.
  • st.warning renders Markdown, so an agent key in a shared report could embed a remote image or a link.
  • CSV formula escaping now also covers full-width ＝＋－＠ and whitespace-prefixed cells.
• 90428c8: the launcher no longer runs code from, or reads settings in, the user's current directory.
  • It runs python -P from a dedicated dashboard_ui/ entry-script directory. Before, a streamlit.py or pandas.py in the current directory was executed, skillevaluator/reporting/html.py shadowed stdlib html inside the server, and a project .streamlit/config.toml could switch off CORS.
  • The Streamlit flags it now sets: showEmailPrompt=false (the old headless=false made a first run without a TTY exit 255 at the email prompt), fileWatcherType=none, toolbarMode=viewer, and allowed hosts limited to 127.0.0.1 and localhost.
• 206c4c1: SIGTERM and SIGHUP are forwarded to Streamlit. Stopping the launcher from an IDE or with kill used to leave the server orphaned on the port.

UX

• 689e4e5: new Comparison reason column, which says, for example, "without-skill arm status failed" or "different cases or attempts". The status filter now narrows only the performance table, like the condition filter. Before, filtering by status turned a failed pair into "missing condition".
• 0842e3d: empty states now say when filters exclude everything, and a hint appears when no pair is comparable. A notice explains that no command records harness_features yet. Coverage text pluralizes "trial".
• 2ca1739: per-trial token, time and cost columns, so rows with different trial counts can be compared.
• 73bf1e6: without --port, Streamlit takes the next free port, so a second dashboard no longer fails on 8501. An explicit busy --port prints a hint.

Install and docs

• 9fd8f57: skillevaluator is not on PyPI, so pip install "skillevaluator[dashboard]" fails, and the name could later be registered by someone else (dependency confusion). The error and docs now give the repo's uv tool install … "skillevaluator[all,dashboard] @ git+…" form, and dashboard is listed in the installation extras table (all does not include it).
• caeffe0:
  • Dropped the two Unreleased Fixed bullets. Token omission shipped in v0.5.0 via feat(tier3): upgrade Harbor to 0.24.0 #83, and the dashboard itself is new.
  • Corrected the reports guide: pairing rule, time and cost sources, multi-step behaviour since v0.5.0, the agent_result token fallback, and the accepted harness_features forms and their precedence.

Hygiene

• 96ba54e: ruff format on dashboard_app.py.
• 3367504: tidy the retained-run loader with no behaviour change:
  • named bounds that mirror report_data's 512/4096 trial limits, pinned by a test;
  • a dead re-check and a double stat removed;
  • shared notice constants;
  • expected made a required keyword;
  • new tests for the trial-discovery limit and the 16 MiB local-file notice.
• 878dd78: tests:
  • encoding="utf-8" on all 48 test file reads and writes, for the Windows job;
  • a 60 s AppTest ceiling with pandas and pyarrow imported first (a cold xdist run had two AppTest failures at 15 s);
  • dashboard is now pinned under Core workflows in --help.

Deferred, not in this PR

• Adverse-vs-without comparisons: no producer emits adverse arms yet.
• Producers writing cost and harness_features.
• Aggregating across runs, and charts.
• A single upload/file size limit: is 16 MB vs 16 MiB intentional?

A full inline review of the whole PR follows separately.

[rng1995]

Automated review: 14 findings (0 P1, 5 P2, 9 P3); 14 posted inline.

[rng1995]

Review follow-up

I posted 14 verified inline findings (5 P2, 9 P3). All are fixed, replied to, and resolved, one commit each (the two docs findings share one). Head is now 60f62a9. The full local suite gives 11,874 passed; the 2 sidecar tests that fail on main in my environment still fail.

• P2 9d2a16c: in large retained runs, non-best agents kept 0 trials because they went through the 2 MiB embedded-report pruning. agent_eval_result_from_directory now takes enforce_embedded_size, which defaults to on, and the dashboard turns it off for this read-only use. Pruned canonical JSON still fails closed, but it now shows the declared trial count and says why.
• P2 f79c9d1: an unreadable subdirectory crashed the whole page. The directory walk now skips it with a notice, and main() turns any per-input error into a notice.
• P2 da81f34: the allowed-hosts protection only exists from Streamlit 1.61, so the extra now requires streamlit>=1.61,<2. A test checks that every flag the launcher passes exists in the installed Streamlit.
• P2 244477c: real runs embed run_config in result.json, so the hand-edited run_config.json that the docs describe was ignored. The file now takes precedence.
• P2 ea19d4a: exports from before 0.5.0 that recorded unmeasured token counters as zero no longer produce token deltas. Those zeros are treated as unavailable, with a notice.
• P3:
  • 6a2bc7c: the coverage denominator no longer counts missing trials twice.
  • 5ef4a47: an unreadable trajectory.json now drops only that trial's tokens; its time and cost are kept.
  • 09cea2b: when a file and its directory are both given as inputs and the file is rewritten, the new content now replaces the cached copy.
  • b8285c5: when Streamlit dies from a signal, the launcher exits with 128 plus the signal number.
  • 60f62a9: SIGINT sent only to the launcher is now passed on to Streamlit.
  • 0ae439c, 4badacf: two tests were tightened.
  • b67c297: two docs and CHANGELOG corrections.

@chrisknvidia, three decisions are yours:

• Please confirm the repeated-attempt pairing change (a6084f0).
• Please confirm the new enforce_embedded_size keyword on the Tier 3 builder.
• Is the 16 MB upload limit vs the 16 MiB local-file limit intentional?