Repository navigation
MNT: Restore KB raw benchmark matrices, aliases, and generator script (PR 1/2) - #742
busysofa15 wants to merge 3 commits into
Conversation
- _raw/: 6 verified benchmark matrices + benchmark_aliases.json (114 mappings) - regenerate_kb.py: within-modality tie-aware percentile -> kb_score, bootstrap CI (seed 42, B=2000, per-key seeded); BOND failures take worst rank; Sub-* subsequence variants scored distinctly; rollup = pooled-sample mean; coverage/eligible/failure/missing counts - _raw/kb_scores.json: 65 canonical detectors x 4 modalities (31 pyod-shipped + externals), 30 unbenchmarked - test_kb_routing_golden.py + fixture: byte-identical (sort_keys=False) guard on the legacy get_kb_for_routing view Codex code-review: 2 rounds, all findings resolved (0 High outstanding). algorithms.json and ad_engine.py untouched; the two-view API is step 5.
Add scoring_view="soft_kb" to ADEngine.get_kb_for_routing: it attaches calibrated within-modality scores from _raw/kb_scores.json and re-sorts by them. The legacy default view stays byte-identical to v3.5.3 (locked by test_kb_routing_golden.py); soft_kb is strictly additive and opt-in. - ad_engine.py: scoring_view param, _attach_soft_kb + _KB_MODALITY map; per-detector kb_* fields incl. kb_score_rank (1-based dense rank, None for unscored); top-level kb_version + kb_scoring_method. - knowledge/__init__.py: KnowledgeBase.kb_scores / kb_scores_meta / get_kb_scores, lazy-loaded, all-null graceful degradation if absent. - pyproject.toml: ship _raw/*.json in the wheel. - regenerate_kb.py + kb_scores.json: _meta gains kb_version, kb_scoring_method (byte-stable regeneration). - test_kb_routing_soft_kb.py: soft-KB attachment, dense rank, image all-null, within-modality, and missing-file degradation. Reviewed by Codex over 2 rounds (converged, no High; schema matches PLAN-kb-redesign.md). Tests: 11 (golden + soft_kb), 55 (KB-router surface).
…g prompt build_routing_prompt now renders the calibrated within-modality kb_score, its bootstrap interval, modality rank, support count, metric scope, and fallback level when the context carries the soft-KB view (kb_version), plus an actionable-uncertainty guide (overlapping intervals -> evaluate a set; low support / fallback -> weaker evidence; within-modality only). Without this the LLM agent never sees the calibrated scores, so the Phase B soft-KB condition (C3) would read the same legacy citation rank as the old KB. - _llm.py: opt-in _build_soft_kb_prompt + _SOFT_KB_GUIDE; requested detector count capped at min(top_k, n_available) so image/text (2 candidates) no longer ask for 3. Legacy default prompt stays byte-identical. - test_kb_routing_soft_kb.py: soft prompt renders scores + guidance, exact row-format lock, legacy prompt has no soft fields, unscored marked "no KB score", and the count cap. Reviewed by Codex over 2 rounds (converged, no High; legacy prompt byte-identical to HEAD for tabular/text/time_series). Tests: 59 (golden + soft-KB + router-surface).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77fc8af959
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| kb_mod = self._KB_MODALITY.get( | ||
| str(target_modality).lower(), str(target_modality).lower()) | ||
| for d in available: | ||
| roll = self.kb.get_kb_scores(d['name'], kb_mod) |
There was a problem hiding this comment.
For a strict data_type="text" request, the available shipped detectors are EmbeddingOD and MultiModalOD, but the NLP benchmark artifact stores scores under the component detector names (LOF, DeepSVDD, ECOD, etc., with per-embedding details). Looking up only d['name'] therefore returns no rollup for every text candidate, so the new soft-KB text routing path reports all detectors as unscored despite shipping NLP benchmark scores. The text path needs to expose/map the scored embedding-detector configurations instead of querying only the wrapper name.
Useful? React with 👍 / 👎.
| [tool.setuptools.package-data] | ||
| "pyod.utils.model_analysis_jsons" = ["*.json"] | ||
| "pyod.utils.knowledge" = ["*.json"] | ||
| "pyod.utils.knowledge" = ["*.json", "_raw/*.json"] |
There was a problem hiding this comment.
This package-data entry includes only JSON files under _raw, while MANIFEST.in also includes only pyod/utils/knowledge/*.json. Consequently none of the newly added benchmark CSVs are selected for an sdist, even though the generator imports them by relative path; users unpacking a source distribution cannot reproduce kb_scores.json. Add the CSVs to the source manifest without adding them to wheel package data if the intended policy is sdist-only.
Useful? React with 👍 / 👎.
| m.pop("by_embedding") | ||
| node["metrics"][metric] = {**summ, **( | ||
| {"by_embedding": m["by_embedding"]} if "by_embedding" in m else {})} | ||
| pooled_all.extend(pooled) |
There was a problem hiding this comment.
When a modality has multiple metrics or embeddings, this flattening discards dataset identity before _summ resamples individual values. Thus correlated measurements from one dataset are treated as independent bootstrap observations and are also counted separately as coverage: for example, 57 ADBench datasets become coverage_n=114 after pooling ROC and PR, while the routing prompt describes n as the number of supporting datasets; text scores similarly report 32 observations from only 8 datasets across two metrics and two embeddings. This overstates support and can make rollup intervals artificially narrow; retain dataset IDs and cluster-resample/count unique datasets for rollups.
Useful? React with 👍 / 👎.
| "n_scored_canonical": 65, | ||
| "n_unbenchmarked": 30 |
There was a problem hiding this comment.
The checked-in artifact reports 30 unbenchmarked detectors, but the unchanged algorithms.json contains 62 keys, only 31 of which appear in scores; AudioAE is absent from both scores and the unbenchmarked array, so the generator's canonical_keys - scores logic would produce 31 entries. This means running the newly supplied generator against this commit immediately changes its claimed generated output, and consumers of the artifact's unbenchmarked inventory silently lose AudioAE; regenerate the JSON so the list and metadata match the current catalog.
Useful? React with 👍 / 👎.
| "n_scored_canonical": 65, | ||
| "n_unbenchmarked": 30 |
There was a problem hiding this comment.
The current algorithms.json contains 62 keys, of which only 31 occur in scores, so this generated count should be 31 rather than 30. The corresponding unbenchmarked array omits AudioAE even though that detector existed before this commit and has no score; running the committed generator against the committed inputs will therefore immediately change this supposedly reproducible artifact, and consumers enumerating unbenchmarked detectors receive incomplete data.
Useful? React with 👍 / 👎.
2 participants
Context
Upstream branch feature/kb-redesign-v3.6 contained 13 orphaned benchmark data files and the generation pipeline required to reproduce the detector ranks in algorithms.json. This PR is part 1 of a 2-PR sequence to restore the full Knowledge Base system without altering runtime routing behavior yet.
Changes in this PR
Restores raw benchmark CSVs (ADBench, BOND, NLP-ADBench, TSB-AD) under pyod/utils/knowledge/_raw/.
Restores benchmark_aliases.json and the generated kb_scores.json.
Restores regenerate_kb.py to allow offline ordinal calibration and bootstrap CI calculations across datasets.
Updates pyproject.toml package-data settings to keep CSVs in sdist while bundling .json files in wheels.
Cleaned up stale docstring references to non-existent scratch files.
Note: Routing logic adjustments and golden test suites will follow in PR 2.