Skip to content

fix(mcp): don't route every "current"/"supported" question to release… - #304

Open
shrxyas-246 wants to merge 1 commit into
kubeflow:mainfrom
shrxyas-246:fix/router-current-supported
Open

shrxyas-246 wants to merge 1 commit into
kubeflow:mainfrom
shrxyas-246:fix/router-current-supported

Conversation

@shrxyas-246

Copy link
Copy Markdown

Fixes #289

cc @kmr-rohit, as we discussed at the Nov 15 community meetup.

What was wrong

pick_search_plan sends a question down the "latest release" path if it contains current or supported. That path only searches release notes, sorts them by date and skips the reranker. So normal how-to questions that just happen to use one of those words get release notes back instead of the docs page with the answer. Two of them are already in the eval set (fid-parallelfor-v2 and dbg-external-mysql).

Question Before After
Is dsl.ParallelFor supported in KFP v2? temporal, release notes only exact
Is running Kubeflow Pipelines with an external MySQL supported, and how is it configured? temporal, release notes only conceptual
How do I get the current run ID inside a KFP component? temporal, release notes only conceptual
Which hyperparameter search algorithms are supported by Katib? temporal, release notes only general
What is the latest Kubeflow release? temporal temporal
What is the current version of Kubeflow Pipelines? temporal temporal

What I changed

  • latest, newest and most recent still always mean the latest release.
  • current and supported only count when they're within 3 words of version or release, in either order. "current version of KFP", "supported Kubernetes versions" and "Which release is current?" still go to release notes, "the current run ID" doesn't.
  • The check is a small helper, asks_for_latest_release(), next to the existing regex. I didn't edit the regex itself because feat(rag): recurring ingest, community-distribution docs, and latest-query retrieval #290 changes that same line, so the two PRs don't touch the same lines and can be merged in either order. The helper only filters out current / supported, so anything else the regex matches still counts, including the "new / next / last release" phrasing feat(rag): recurring ingest, community-distribution docs, and latest-query retrieval #290 adds.
  • Updated the temporal row in docs/RAG_V4_ARCHITECTURE.md to match.
  • Added tests/test_intent_router_temporal.py: 6 release questions that must stay temporal and 6 how-to questions that must not, including the keyword forms the system prompt asks the model to send.

Results

Before (main): all four how-to questions go to release notes and the 6 how-to test cases fail.

before, on main

After (this PR): the how-to questions get normal docs search and the release questions are unchanged. 12 passed, full suite 172 passed, 69 skipped (the 160 existing tests plus the 12 new ones).

after, this PR

On top of #290: the same change applies cleanly to #290's branch. 12 passed, full suite 202 passed, 69 skipped.

on top of PR 290

… notes

The router sent any docs query containing "current" or "supported" down
the latest-release path: release notes only, sorted by date, no reranker.
How-to questions like "Is dsl.ParallelFor supported in KFP v2?" or "How do
I get the current run ID inside a KFP component?" got release notes back
instead of the docs pages that answer them.

"latest", "newest" and "most recent" still always count. "current" and
"supported" now only count within three words of "version" or "release",
so "current version of KFP", "supported Kubernetes versions" and "Which
release is current?" still search release notes.

The check is a small helper next to the existing regex rather than a
change to the regex itself, so it doesn't touch the lines kubeflow#290 changes.

Fixes kubeflow#289

Signed-off-by: Shreyas <shreyassrini2007@gmail.com>
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tarekabouzeid for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(mcp): "current" and "supported" send how-to questions to release-notes-only search

1 participant