You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug(pipelines): incremental-pipeline.py is broken and diverged from RAG v4 hybrid schema #305
With the introduction of RAG Architecture v4 (#249), the production documentation pipeline (kubeflow-pipeline.py) was migrated to milvus_store.py and canonical_rag_ingest.py, introducing Lean Schema v4:
Structural metadata (title, section_path, version, doc_type, and epoch release_date).
Contextual breadcrumbs with sentinel delimiter <!--kubeflow:context:end-->.
However, docs-agent-mcp/pipelines/incremental-pipeline.py was left on the legacy v2 architecture:
Schema Mismatch:store_milvus_incremental inserts records using file_unique_id without document_id, sparse_vector, release_date, or section_path. Running this against kubeflow_docs will crash or corrupt the v4 collection.
Broken Deletion Query:delete_old_vectors queries file_unique_id == ..., which does not exist in Schema v4.
Outdated Ingestion & Missing Breadcrumbs:chunk_and_embed_incremental still uses legacy regex stripping and RecursiveCharacterTextSplitter, completely bypassing canonical Hugo parsing and contextual breadcrumb prefixes.
TEI Truncation: Hardcodes 1000-char limits instead of sharing truncate_for_tei() bounds.
Expected behavior
incremental-pipeline.py should reuse shared ingestion and storage utilities (milvus_store.py, canonical_rag_ingest.py), updating changed documentation files into the Schema v4 hybrid collection idempotently.
Proposed Solution:
Refactor chunk_and_embed_incremental to use milvus_store.chunk_github_jsonl() (or canonical_rag_ingest.build_milvus_records) and milvus_store.embed_chunk_records().
Refactor store_milvus_incremental to delegate directly to milvus_store.store_embedded_records(..., clean_rebuild=False), which natively handles document-level chunk replacement and BM25 index updates.
Update delete_old_vectors to use MilvusClient and filter on document_id == f"{repo_name}:{file_path}".
Add unit test coverage in tests/test_incremental_pipeline.py verifying that incremental chunking produces valid v4 records.
Activity
ABHISHEKUPAI commentedon Nov 20, 2026
/assign