Surface knowledge-ingestion runs, search storylines by anchor, and seed issuers locally - #1298
Merged
Merged
Conversation
Three follow-ups to the knowledge graph.
The knowledge-ingestion run counters had no reader, so a run that skipped
half its batch looked identical to a clean one. Add a read-only chronicle at
order 46 showing what each run considered, opened and skipped, with the
developments it wrote linking back to their storylines.
Storyline search only matched the name, so looking for a thread by what it
is about ("contract delay") found nothing unless the phrase happened to be
in the title. Search the anchors too, and add the trigram indexes that make
an unanchored substring match indexable rather than a sequential scan.
The repository ships seed-data/ticker-profiles.json with no loader, so a
fresh local database has no issuers and nothing downstream to hang off. Add
seed-tickers-local to load all 949.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the three gaps left open when the knowledge graph shipped in #1296: the ingestion run counters had no reader, storyline search ignored the anchors, and a fresh local database had no issuers to hang anything off. Everything here was verified against a live stack with real articles rather than fixtures.
Related issues
Closes #1297
Important changes
knowledge-ingestion-runsresource at order 46. The six per-run counters plus a derived attach rate and duration, with the developments each run wrote linking back to their storylines.Development.ingestionRunIdis no longer a dead end. A run that dropped sources for want of anchors now says so in its section badge.storyline.nameor anystoryline_anchor.anchor. Verified live, where "rapid" appears in no storyline name yet correctly returns two storylines.contains, which a btree index cannot serve, so every search degraded to a sequential scan as the knowledge base grew. The migration enablespg_trgmand adds GIN indexes onstoryline.nameandstoryline_anchor.anchor.seed-tickers-local. The repository shipsseed-data/ticker-profiles.jsonwith 949 issuers and no loader, so a fresh database had no tickers. The script upserts by symbol, so re-running only refreshes fields.Other changes
titlefield the run detail mapper computed that nothing rendered.Key files to review
apps/mediapulse/domain-api/src/resources/knowledge-ingestion-runs/dashboard-page.ts- two stat-card rows, the key-value block, and the developments table.packages/mediapulse/database/prisma/migrations/20260916000000_storyline_search_trigram_indexes/migration.sql-CREATE EXTENSIONplus both GIN indexes, allIF NOT EXISTS.apps/mediapulse/domain-api/src/resources/storylines/list-filters.ts- theORacross name and anchors.packages/mediapulse/database/scripts/seed-tickers-local.ts- hand-rolled row validation rather than zod, which is not a dependency of that package.How to test
pnpm code-qualityfrom the repo root. 111 tasks, all green, Prettier clean.docker compose up -d postgres, thenpnpm --filter @mediapulse/database db:migrate:deploy.pnpm --filter @mediapulse/database run seed-tickers-localand confirm 949 issuers, then run it again and confirm it is a no-op./dashboard/mediapulse/knowledge-ingestion-runsand/dashboard/mediapulse/storylines.Note the agent's run route is
POST /, not/runas #1296 stated.