feat(app): make search N-ary, with multi-source as the default shape - #2869
Draft
teeohhem wants to merge 6 commits into
Draft
feat(app): make search N-ary, with multi-source as the default shape#2869teeohhem wants to merge 6 commits into
teeohhem wants to merge 6 commits into
Conversation
The search page's source selector can expand into a multi-select (up to 5 log/trace sources). Each selected source runs its own query pipeline (own connection, Lucene serializer, windowed pagination) with its SELECT rewritten to canonical aliases; a client-side k-way merge interleaves the streams by timestamp behind a "safe frontier" so the timeline never shows a gap a lagging source could still fill. Results render in one table with per-row source badges, a histogram stacked by source, summed totals, and an add-column picker over the union of the sources' columns. Multi mode is Lucene-only and URL-shareable (?sources=). Saved searches, alerts, filters, and delta/pattern modes stay single-source and are gated off with explanations; single-source behavior is unchanged. Also fixes chSqlToAliasMap dropping NULL-literal projections, which would have made row-WHERE clauses reference nonexistent columns.
Three components were each hand-unrolling slot0..slot4 hook calls to keep a constant hook count across a variable source selection. Rules of hooks allow calling a use*-named function parameter from a custom hook, so one generic useMultiSourceSlots(items, useSlot, opts) now owns the unrolling and the MAX_SEARCH_SOURCES pin; consumers are a single call. Slot hooks return memoized values so the returned array is dependency-list safe.
Query cost scales linearly with the selection (~3 ClickHouse queries per source per refresh, re-fired every live-tail tick, plus one-time metadata), and past 3 sources the interleaved timeline stops being legible. 3 covers the common "app logs + infra logs + traces" case.
The filters sidebar now works across a multi-source selection. Each selected source runs its own facet pipeline (fields + values); the sidebar merges them by field path with values unioned, reusing the single-source FilterGroup rendering. Checking a value filters every source that has the field. A source whose table lacks a filtered column is excluded from the search with a visible reason on its status chip, rather than silently returning rows that ignore the filter. Filter pills and add-to-filter from the row side panel are re-enabled in multi mode, with filter-key escaping and date-column detection driven by the union of the selected sources' schemas.
The search page had two parallel result paths: DBSqlRowTable for a single source and a separate table for several. This collapses them. The N-ary pipeline now handles a single source as the degenerate case — its spec carries the user's own SELECT and ORDER BY, so the table renders the authored columns (no source badge), sorts, denoises, and reports errors to the page exactly as before. Multiple sources still project canonical aliases and merge on the client. Renames the component to SearchResultsTable, since it is now the search page's only results table, and extracts the denoise pipeline and the select-column resolution out of DBRowTable so both callers share one implementation. DBSqlRowTable stays for dashboards, k8s pages, and side panels; only the search page's use of it is gone.
🦋 Changeset detectedLatest commit: 7e4c1d8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Aug 11, 2026
Contributor
Greptile SummarySearch is generalized to query up to three log or trace sources independently and merge their results into a shared timeline.
Confidence Score: 3/5The PR is not yet safe to merge because a failed schema request can suppress every selected source and generated row-side-panel links discard the multi-source selection. A selected source whose schema request fails remains absent from columnsBySourceId, causing the active-filter or extra-column gates to return no query specifications for healthy sources; separately, generateSearchUrl writes only the primary source even though multi-source restoration depends on the sources parameter. Files Needing Attention: packages/app/src/DBSearchPage.tsx, packages/app/src/hooks/useMultiSourceSearch.ts
|
| Filename | Overview |
|---|---|
| packages/app/src/DBSearchPage.tsx | Introduces multi-source selection and query orchestration, but schema failures can still suppress all streams and generated side-panel links still omit the selected source list. |
| packages/app/src/components/SearchResultsTable.tsx | Provides the unified single- and multi-source results table with per-source pagination, status reporting, and merged rendering. |
| packages/app/src/hooks/useMultiSourceSearch.ts | Adds fixed hook slots and schema-union utilities, but schema query errors are discarded and therefore cannot be distinguished from loading. |
| packages/app/src/utils/multiSourceMerge.ts | Implements timestamp ordering, safe-frontier advancement, source tagging, and multi-stream merge behavior. |
| packages/common-utils/src/core/searchChartConfig.ts | Adds canonical per-source search projections while retaining each source’s ClickHouse query configuration. |
Sequence Diagram
sequenceDiagram
participant U as User
participant P as Search page
participant S as Selected sources
participant Q as Per-source queries
participant M as Safe-frontier merge
participant V as Timeline and charts
U->>P: Select 1–3 sources and search
P->>S: Resolve source configurations
loop Each selected source
S->>Q: Build source-specific query
Q-->>M: Return rows and progress frontier
end
M->>M: Interleave rows by timestamp
M-->>V: Emit rows safe to display
Q-->>V: Emit per-source histogram and count data
Reviews (2): Last reviewed commit: "Merge branch 'main' into tom/search-n-ar..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
E2E Test Results✅ All tests passed • 280 passed • 1 skipped • 1231s
Tests ran across 4 shards in parallel. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #2862 and #2865 (both draft, both contained here as commits). Review this one instead — it delivers the same capability, but multi-source is the default shape of search rather than a mode bolted onto it.
Searching has meant picking exactly one source, so a question spanning sources (several log sources, or logs plus traces) takes multiple tabs and manual correlation. Search is now N-ary: pick up to 3 log/trace sources and results interleave into one timestamp-ordered timeline with per-row source badges, normalized columns, a histogram stacked by source, summed totals, merged filter facets, and an add-column picker over the union of the sources' columns.
The important structural point is that one source is not a separate code path — it is N=1. The results table takes a list of sources and, with one, carries that source's own SELECT and ORDER BY, so it renders the authored columns (no badge column), sorts, denoises, and reports errors to the page exactly as it always has. With more than one, each source projects canonical aliases (
__hdx_timestamp,__hdx_severity, ...) derived from its semantic expressions and a client-side k-way merge orders rows behind a "safe frontier" — the timestamp every source has covered — so the timeline never shows a gap a slower source could still fill, and "fetch more" advances only the lagging source. There is noUNION ALL: sources can live on different ClickHouse connections, each keeps its per-table machinery (Lucene serializer, text-index detection, materialized-column rewrites, query settings), and a failing source degrades to a status chip instead of failing the search.Filters are per-source: checking a value filters every source that has the field, and a source whose table lacks a filtered column is excluded with a visible reason rather than silently returning unfiltered rows. Multi mode is Lucene-only for now (raw SQL names one table's columns); saved searches and alerts remain single-source. Selections are shareable via
?sources=.Design notes for reviewers
SearchResultsTable(renamed from the multi-source-only component) is the search page's only results table now.DBSqlRowTablestays for dashboards, k8s pages, and side panels — only search's use of it is gone.DBRowTableinto shared exports so both callers use one implementation.useMultiSourceSlots, which unrolls a fixedMAX_SEARCH_SOURCEShook slots in exactly one documented place.dbSqlRowTablequery-key prefix (so "is the search fetching?" measures what it always did), and the empty-SELECT guard is preserved.Verification
Single-source parity is the bar this has to clear, so: the full
searchE2E folder passes (37 passed, 8 flaky, 0 failed), and single-source was driven manually against the demo stack for columns, sorting, denoise, side panel, and live tail. One test (Should pin filter and verify it persists after reload) fails on this branch's first attempt — it fails identically onorigin/mainand on the superseded branches, so it is pre-existing, not a regression; it is a null-guard bug inusePersonalPinnedFiltersworth a separate fix.How to test on Vercel preview
Preview routes: /search
Steps: