fix: prevent BM25 query token ID overflow

[RAMZI0TO99]

The real token ad1u66pi has mmh3.hash(token) == -2147483648. Taking its absolute value yields ID 2147483648, which the existing int32 query index array cannot store. Under NumPy 2.3.5 a nonempty query raises OverflowError, while the document embedding accepts the ID.

This widens query indices to int64 and preserves the hash function and existing IDs. Tests exercise the actual mmh3 boundary token, matching document/query IDs, normalization, repeated and ordinary tokens, string/list/generator input, and empty queries. They use a local model path with stemming disabled and require no model download. Collision weighting is unchanged.

Validation on Windows 11, Python 3.13.5, NumPy 2.3.5, FastEmbed 0.8.1:

• 11 selected tests passed: python -m pytest -q tests/test_bm25_query_index_overflow.py.
• Local diff-scoped AST docstring coverage: 9/9 functions (100%, above the reported 80% threshold). CodeRabbit's own result is separate.
• Ruff 0.3.4 lint/format and staged whitespace checks passed.
• Repository mypy and pyright checks passed on Python 3.13.5.

The full model/OS/Python CI matrix was not run locally.

Split from #766 following the maintainer's request for one PR per problem. This branch is based directly on current main and contains only this problem's production change and regression module.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 7fd8975d-985f-4a16-b2a7-256d171e90df

📥 Commits

Reviewing files that changed from the base of the PR and between 72f5a24 and 632c77c.

📒 Files selected for processing (1)

• fastembed/sparse/bm25.py

💤 Files with no reviewable changes (1)

• fastembed/sparse/bm25.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

Bm25.query_embed now creates query token ID arrays with the NumPy int64 dtype instead of int32. Query weighting and token deduplication are unchanged.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 632c7

The wider query indices accommodate the reported overflow boundary without changing computed token IDs. No merge-blocking issue was found.

Security Architecture Review

Security architecture risk: ⚪ Minimal · up to 72f5a

The change fixes a boundary-token failure while preserving existing token IDs. The shared sparse-vector interface already accepts int64 indices, and the reviewed paths introduce no new privileges, persistent state, or trust-boundary crossings.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The visible change reaches BM25 query consumers through returned NumPy array dtypes. The newly representable query boundary ID already belongs to the established hash identity space and matches document output; the inspected change does not expand tenant, asset, or privilege scope.

Trust Boundaries and Controls

• inferred — Caller-controlled query text follows the same preprocessing and hash-to-vector path before and after the change. Widening storage removes a representation failure without bypassing an existing control in the inspected path.

Resilience and Maintainability Implications

• inferred — The changed computation uses fresh per-query arrays rather than a shared mutable transition. Repetition, interruption between yields, or failure during a query therefore introduces no new reservation, persistent partial state, or recovery obligation in this method.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the BM25 query token ID overflow fix, which is the main change.
Description check	✅ Passed	The description explains the overflow issue, the int64 change, and the regression tests, so it is directly related to the changeset.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 2 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[joein]

is that a real-real token ad1u66pi ? 😄

[RAMZI0TO99]

It's an alphanumeric boundary-case test input. I selected it for mmh3.hash("ad1u66pi") == -2147483648, so abs(...) becomes 2147483648. I haven't observed it in a production corpus. Thanks for reviewing and merging the fix!