Skip to content

Scope repository-relative knowledge ingestion to configured paths - #17256

Open
Jialin Huang (JialinHuang803) wants to merge 1 commit into
mainfrom
jialinhuang/knowledge-path-scoping
Open

Jialin Huang (JialinHuang803) wants to merge 1 commit into
mainfrom
jialinhuang/knowledge-path-scoping

Conversation

@JialinHuang803

Copy link
Copy Markdown
Member

Summary

Generic, independently main-based knowledge-sync fix; no .NET source configuration or agent additions. Companion #17253 must merge after this PR.

Previously, relativeByRepoPath: true used the configured path for sparse checkout but discarded it during ingestion. Every entry scanned the entire shared sparse-checkout repository root. Entries sharing a destination folder could produce duplicate blob keys and overwrite each other's plane metadata.

  • Preserve the repository root for blob filenames/citations while limiting ingestion to the configured directory or individual Markdown/MDX file.
  • Keep pathless entries scanning the entire available repository checkout and preserve ordinary non-relative directory/file behavior and existing filters/metadata resolution.
  • Check the actual selected path before processing. Restrict existing root release-notes index discovery to selections containing it; do not introduce nested release-note discovery.
  • Add synthetic, hermetic regressions for selection roots and sparse-checkout paths, disjoint same-folder entries and independent plane metadata, duplicate-basename file collisions, Markdown/MDX and nested references, filters, missing selections, and release-note guards. Export processing helpers for local filesystem tests without Azure access.

Intentional global ingestion and cleanup impact

This behavior change applies to all relativeByRepoPath sources, not only the companion .NET sources. Root/ancestor Markdown incidentally fetched by sparse checkout but outside configured paths will no longer ingest. On a later authorized sync, cleanupExpiredBlobs can remove previously indexed incidental blobs absent from the scoped results. Review the complete configuration and migration impact before syncing. Selected paths sharing a destination folder must not overlap, especially with different metadata.

No live sync, indexing, deployment, or cloud changes were performed. Never run sync against a partial source list: existing cleanup removes blobs absent from the full processing results. All existing configuration and dependency manifests/locks remain untouched.

Validation

  • Targeted configuration/path-scoping suite: 14 tests passed.
  • Full knowledge-sync suite: 103 tests passed across 15 files.
  • npm run build: passed.
  • Scoped staged diff check: passed.

Initial tests/build failed because dependencies were absent; restored existing locked dependencies with npm ci without changing manifests/locks. Existing unrelated Windows long-path checkout omissions were left untouched and are not part of this PR.

Preserve repository-relative blob identities while avoiding repeated shared-root scans. Add hermetic generic regressions and document global ingestion and cleanup migration impact.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
71 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Missing selections can trigger destructive cleanup, while selected-file filters and backslash paths are handled incorrectly.

3 open findings
What changed in this PR

Scopes repository-relative knowledge ingestion to configured paths while preserving repository-root citation identities.

Changes:

  • Adds scoped directory/file processing and release-note guards.
  • Adds regression coverage for path selection, metadata, filters, and collisions.
  • Documents cleanup and migration implications.
File Description
src/​DailySyncKnowledge.ts Scopes ingestion and release-note discovery.
src/​services/​ConfigurationLoader.ts Adds selected-path tracking.
tests/​sourcePathScoping.test.ts Adds path-scoping regressions.
tests/​configurationLoader.test.ts Tests source transformation.
README.md Documents behavior and migration impact.

🧠 Review effort: Balanced

Comment on lines +107 to 110
if (!fs.existsSync(scanPath)) {
console.warn(`Source path not found: ${scanPath}`);
continue;
}
Comment on lines 430 to +431
if (isFile) {
processSingleFile(sourceDir, path.dirname(sourceDir));
processSingleFile(scanPath, source.includePath ? sourceDir : path.dirname(sourceDir));
for (const docPath of source.paths) {
sources.push({
path: (docPath.relativeByRepoPath || docPath.path === undefined) ? `docs/${repoPath}` : `docs/${repoPath}/${docPath.path}`,
includePath: docPath.relativeByRepoPath ? docPath.path?.replace(/^[/\\]+/, '') : undefined,

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants