Skip to content

fix(pipelines): align incremental-pipeline.py with canonical RAG v4 schema and ingestion - #306

Open
ABHISHEKUPAI wants to merge 1 commit into
kubeflow:mainfrom
ABHISHEKUPAI:fix-incremental-pipeline-v4
Open

ABHISHEKUPAI wants to merge 1 commit into
kubeflow:mainfrom
ABHISHEKUPAI:fix-incremental-pipeline-v4

Conversation

@ABHISHEKUPAI

@ABHISHEKUPAI ABHISHEKUPAI commented Nov 20, 2026 •

Copy link
Copy Markdown

Summary

fixes #305

This PR ports the incremental documentation ingestion pipeline (incremental-pipeline.py) to the canonical RAG Architecture v4 (milvus_store.py and canonical_rag_ingest.py).

Following the initial RAG v4 hybrid migration (#249), kubeflow-pipeline.py was switched to Lean Schema v4 with native Milvus BM25, Hugo-aware parsing, and structured metadata. However, incremental-pipeline.py was left on the legacy v2 architecture:

  1. It inserted deprecated Schema v2 records using file_unique_id without document_id, sparse_vector, release_date, doc_type, or section_path, which fails or corrupts the production kubeflow_docs v4 collection.
  2. The vector deletion step queried file_unique_id == ..., which does not exist in Schema v4.
  3. Chunks were created using legacy regex stripping and RecursiveCharacterTextSplitter, bypassing canonical Hugo block parsing.

Changes Made

  1. Aligned Ingestion & Embeddings (docs-agent-mcp/pipelines/incremental-pipeline.py):
    • Replaced legacy text cleaning and RecursiveCharacterTextSplitter with milvus_store.chunk_github_jsonl(...) and milvus_store.embed_chunk_records(...).
    • Changed default parameters from character-based (chunk_size=1200, chunk_overlap=100) to token-based (target_tokens=DEFAULT_TARGET_TOKENS, overlap_tokens=DEFAULT_OVERLAP_TOKENS) from milvus_store.
    • Ensured safe handling of empty dataset inputs when only file deletions occur.
  2. Aligned Storage & Deletion (delete_old_vectors & store_milvus_incremental):
    • Migrated delete_old_vectors to MilvusClient and updated the deletion filter to document_id in [...] (using repo_name:file_path).
    • Refactored store_milvus_incremental to delegate directly to milvus_store.store_embedded_records(..., clean_rebuild=False), which provides native document-level replacement and BM25 indexing.
    • Guarded against empty input files to avoid no-op errors during pure deletion runs.
  3. Documentation (docs-agent-mcp/pipelines/README.md):
    • Updated the incremental pipeline documentation to reflect that it targets the v4 hybrid collection.
  4. Unit Tests (tests/test_incremental_pipeline.py):
    • Added a test suite with 7 unit tests covering:
      • Filtering non-documentation files in download_specific_files.
      • delete_old_vectors querying by document_id in [...] via MilvusClient.
      • Graceful handling of empty or invalid JSON file paths in delete_old_vectors.
      • chunk_and_embed_incremental generating valid Schema v4 records with titles and section paths.
      • Empty dataset handling in chunk_and_embed_incremental.
      • store_milvus_incremental delegating to store_embedded_records(clean_rebuild=False).
      • store_milvus_incremental cleanly skipping empty input files.

Verification

  • Ran the new test suite via pytest:
    pytest tests/test_incremental_pipeline.py -v

…schema and ingestion

Signed-off-by: ABHISHEKUPAI <abhishekupai6055@gmail.com>
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chasecadet for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(pipelines): incremental-pipeline.py is broken and diverged from RAG v4 hybrid schema

1 participant