feat(mcp): improve metric discovery, add quiet-saturation eval scenario - #2861
feat(mcp): improve metric discovery, add quiet-saturation eval scenario#2861karl-power wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 8a7ad67 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Additional context: 8 file(s) in private internal-tooling packages, excluded from the line count Review process: Full human review — logic, architecture, edge cases. Stats
|
Greptile SummaryThe PR makes metric names discoverable directly from the MCP source catalog and adds a quiet-saturation evaluation scenario.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| packages/api/src/mcp/tools/sources/listSources.ts | Adds team-scoped, timeout-bounded metric preview collection and direct-query guidance to the source catalog. |
| packages/api/src/mcp/tools/sources/metricNames.ts | Extracts metric-name sampling into a shared helper and widens sparse-metric discovery from 24 hours to 30 days. |
| packages/api/src/mcp/tools/sources/describeSource.ts | Replaces the local metric sampler with the shared widening-lookback implementation. |
| packages/hdx-eval/src/grading/programmatic.ts | Evaluates informational adoption checks while excluding their zero weight from adoption scoring. |
| packages/hdx-eval/src/reports/aggregate.ts | Propagates informational adoption metadata into aggregate reporting. |
| packages/hdx-eval/src/reports/markdown.ts | Labels informational checks and explains their exclusion from adoption scores. |
| packages/hdx-eval/src/scenarios/quiet-saturation/generate.ts | Generates deterministic telemetry for diagnosing a gradual connection-pool leak and its distractors. |
| packages/hdx-eval/src/scenarios/quiet-saturation/ground-truth.json | Defines outcome, judge, and metric-adoption criteria for the new scenario. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[clickstack_list_sources] --> B[Load team sources and connections]
B --> C[Build source summaries]
C --> D[Sample metric tables concurrently]
D --> E[Try 24-hour lookback]
E -->|Empty| F[Try 30-day lookback]
E -->|Names found| G[Attach metricNamesPreview]
F -->|Names found| G
F -->|Empty or timeout| H[Return summary without preview]
G --> I[Return catalog and metrics usage guidance]
H --> I
Reviews (3): Last reviewed commit: "feat(mcp): improve metric discovery, add..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Deep Review✅ No critical issues found. The change is defensively built: metric-name preview sampling in No P0/P1/P2 findings with a concrete failure mode surfaced from the completed analysis. 🔵 P3 nitpicks (1)
Reviewers (1): security (completed, no findings). The full panel (correctness, performance, reliability, adversarial, testing, maintainability, api-contract, project-standards, agent-native, learnings) was dispatched but had not returned at synthesis time; the assessment above reflects the security reviewer plus direct analysis of every changed production file ( Testing gaps:
Residual risk: under a quiet environment where the 24h window is empty across many metric sources, every kind falls through to a 30d scan; the client-side 3s budget bounds |
E2E Test Results✅ All tests passed • 280 passed • 1 skipped • 1093s
Tests ran across 4 shards in parallel. |
2e3c3f7 to
3b1a262
Compare
3b1a262 to
8a7ad67
Compare
1 participant
Why
Eval transcripts showed agents solving investigations without ever touching metrics — not because metrics weren't useful, but because they were the only signal behind a discovery wall. After
list_sources, logs and traces are queryable immediately (key columns are in the catalog), while a metric source showed only opaque table names: querying it cost 1–3 extra calls (describe_source/list_metrics) just to learn what exists. Under a turn budget, agents rationally skipped it — even when explicitly nudged (we tested this: prompt- and output-level hints naming the exact metrics were ignored in 7/7 deliveries when metrics weren't on the efficient path).What changed
clickstack_list_sources: metric sources now includemetricNamesPreview— up to 10 recently-reported metric names per kind, sampled from the team's own tables — plus a usage note that metrics are queried directly viaclickstack_table/clickstack_timeserieswithmetricType + metricName, no describe hop needed. Best-effort under a 3s wall-clock budget with a concurrency pool and per-table dedup; omitted silently on timeout.metricNames.ts(new): the metric-name sampler extracted fromdescribeSource.tsinto a shared module, with a widening lookback (24h → 30d, first non-empty window wins) so sparse or batch-emitted metrics still produce a sample.describe_sourceuses the same sampler, so its per-kind sample gets the lookback too.Measured wins
Benchmarked with the hdx-eval framework on
quiet-saturation(connection-pool leak where metric history is the efficient diagnostic path), branch vsmain, identical seed/anchor/prompts, claude-fable-5 + claude-opus-4-6, 3 runs/cell, two independent batches.First tool call whose args name a target metric (the load-bearing pool gauges), every run, both batches:
Zero overlap across all 24 runs (branch ≤7, main ≥8; p ≈ 0.001 by permutation): the preview removes the discovery hops entirely (
list_sources→ query), where main insertslist_metrics/describe_sourcedetours or trace-grinds first.Headline metrics from the second batch (which ran exactly this minimal build):
distinguishes_true_onset)