Skip to content

Surface knowledge-ingestion runs, search storylines by anchor, and seed issuers locally - #1298

Merged
kevinhermawan merged 1 commit into
mainfrom
knowledge-graph-followups
Sep 16, 2026
Merged

kevinhermawan merged 1 commit into
mainfrom
knowledge-graph-followups

Conversation

@kevinhermawan

@kevinhermawan kevinhermawan commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes the three gaps left open when the knowledge graph shipped in #1296: the ingestion run counters had no reader, storyline search ignored the anchors, and a fresh local database had no issuers to hang anything off. Everything here was verified against a live stack with real articles rather than fixtures.

Related issues

Closes #1297

Important changes

  • knowledge-ingestion-runs resource at order 46. The six per-run counters plus a derived attach rate and duration, with the developments each run wrote linking back to their storylines. Development.ingestionRunId is no longer a dead end. A run that dropped sources for want of anchors now says so in its section badge.
  • Storyline search reads the anchors, not just the name. Searching for what a thread is about now finds it: the term matches storyline.name or any storyline_anchor.anchor. Verified live, where "rapid" appears in no storyline name yet correctly returns two storylines.
  • Trigram indexes for that search. Both columns are searched with an unanchored case-insensitive contains, which a btree index cannot serve, so every search degraded to a sequential scan as the knowledge base grew. The migration enables pg_trgm and adds GIN indexes on storyline.name and storyline_anchor.anchor.
  • seed-tickers-local. The repository ships seed-data/ticker-profiles.json with 949 issuers and no loader, so a fresh database had no tickers. The script upserts by symbol, so re-running only refreshes fields.

Other changes

  • Three shared list-filter definitions for the run status dropdown and the storyline filters.
  • Removed a title field the run detail mapper computed that nothing rendered.

Key files to review

  • apps/mediapulse/domain-api/src/resources/knowledge-ingestion-runs/dashboard-page.ts - two stat-card rows, the key-value block, and the developments table.
  • packages/mediapulse/database/prisma/migrations/20260916000000_storyline_search_trigram_indexes/migration.sql - CREATE EXTENSION plus both GIN indexes, all IF NOT EXISTS.
  • apps/mediapulse/domain-api/src/resources/storylines/list-filters.ts - the OR across name and anchors.
  • packages/mediapulse/database/scripts/seed-tickers-local.ts - hand-rolled row validation rather than zod, which is not a dependency of that package.

How to test

  1. pnpm code-quality from the repo root. 111 tasks, all green, Prettier clean.
  2. docker compose up -d postgres, then pnpm --filter @mediapulse/database db:migrate:deploy.
  3. pnpm --filter @mediapulse/database run seed-tickers-local and confirm 949 issuers, then run it again and confirm it is a no-op.
  4. Start agent-auth-api (8080), agent-data-api (8081), agent-registry-api (8082), domain-api (8090) and Hermes (3001).
  5. Open /dashboard/mediapulse/knowledge-ingestion-runs and /dashboard/mediapulse/storylines.
  6. Search the storylines list for a term that appears only in an anchor and confirm the storyline comes back.

Note the agent's run route is POST /, not /run as #1296 stated.

Three follow-ups to the knowledge graph.

The knowledge-ingestion run counters had no reader, so a run that skipped
half its batch looked identical to a clean one. Add a read-only chronicle at
order 46 showing what each run considered, opened and skipped, with the
developments it wrote linking back to their storylines.

Storyline search only matched the name, so looking for a thread by what it
is about ("contract delay") found nothing unless the phrase happened to be
in the title. Search the anchors too, and add the trigram indexes that make
an unanchored substring match indexable rather than a sequential scan.

The repository ships seed-data/ticker-profiles.json with no loader, so a
fresh local database has no issuers and nothing downstream to hang off. Add
seed-tickers-local to load all 949.
@kevinhermawan
kevinhermawan merged commit 470a300 into main Sep 16, 2026
25 checks passed
@kevinhermawan
kevinhermawan deleted the knowledge-graph-followups branch September 16, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface knowledge-ingestion runs, search storylines by anchor, and seed issuers locally

2 participants