Skip to content

bug(pipelines): incremental-pipeline.py is broken and diverged from RAG v4 hybrid schema #305

Open

Description

@ABHISHEKUPAI

Area

pipelines

What happened

With the introduction of RAG Architecture v4 (#249), the production documentation pipeline (kubeflow-pipeline.py) was migrated to milvus_store.py and canonical_rag_ingest.py, introducing Lean Schema v4:

  • Primary key document_id (replacing legacy file_unique_id).
  • Native Milvus BM25 sparse vectors (FunctionType.BM25).
  • Structural metadata (title, section_path, version, doc_type, and epoch release_date).
  • Contextual breadcrumbs with sentinel delimiter <!--kubeflow:context:end-->.
    However, docs-agent-mcp/pipelines/incremental-pipeline.py was left on the legacy v2 architecture:
  1. Schema Mismatch: store_milvus_incremental inserts records using file_unique_id without document_id, sparse_vector, release_date, or section_path. Running this against kubeflow_docs will crash or corrupt the v4 collection.
  2. Broken Deletion Query: delete_old_vectors queries file_unique_id == ..., which does not exist in Schema v4.
  3. Outdated Ingestion & Missing Breadcrumbs: chunk_and_embed_incremental still uses legacy regex stripping and RecursiveCharacterTextSplitter, completely bypassing canonical Hugo parsing and contextual breadcrumb prefixes.
  4. TEI Truncation: Hardcodes 1000-char limits instead of sharing truncate_for_tei() bounds.

Expected behavior

incremental-pipeline.py should reuse shared ingestion and storage utilities (milvus_store.py, canonical_rag_ingest.py), updating changed documentation files into the Schema v4 hybrid collection idempotently.

Proposed Solution:

  1. Refactor chunk_and_embed_incremental to use milvus_store.chunk_github_jsonl() (or canonical_rag_ingest.build_milvus_records) and milvus_store.embed_chunk_records().
  2. Refactor store_milvus_incremental to delegate directly to milvus_store.store_embedded_records(..., clean_rebuild=False), which natively handles document-level chunk replacement and BM25 index updates.
  3. Update delete_old_vectors to use MilvusClient and filter on document_id == f"{repo_name}:{file_path}".
  4. Add unit test coverage in tests/test_incremental_pipeline.py verifying that incremental chunking produces valid v4 records.

Activity

  1. ABHISHEKUPAI commented on Nov 20, 2026

    @ABHISHEKUPAI
    Author

    /assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions