Repository navigation
Description
Describe the bug
The TopPSampler documentation and its implementation describe opposite rules for which documents are kept.
The TopPSampler docs page says:
The practical goal of the TopPSampler is to return a list of documents that, in sum, have a score larger than the top_p value.
But TopPSampler.run() in haystack/components/samplers/top_p.py selects documents with:
condition = (cumulative_probs <= top_p) | close_to_top_pThe docs describe keeping documents until their cumulative probability exceeds top_p. The code keeps documents only while their cumulative probability stays at or below top_p, so the document that would push the total above top_p is dropped.
On the example from the docs page, the two rules return different documents (see To Reproduce).
Error message
No error is raised. The component runs normally and silently returns a different set of documents than the documentation describes.
Expected behavior
Following the docs, TopPSampler(top_p=0.99) on the docs page example should return Sarajevo and Belgrade (cumulative probability 0.9976). The code returns only Sarajevo (0.9842).
It's unclear to me which behaviour is intended, so either:
- the docs should describe the current rule (keep documents while their cumulative probability stays at or below
top_p), or - the code should follow the documented rule.
For reference, standard top-p sampling (Holtzman et al., 2019) keeps the smallest set whose cumulative probability reaches p, which matches the docs rather than the code.
Additional context
The example in the TopPSampler docstring uses top_p=0.95 and expects only Sarajevo. Under the current rule, no document is selected at all, because Sarajevo alone already has a cumulative probability of 0.9842, above 0.95. Sarajevo is returned only through the fallback, which logs a warning (see output below). So the docstring example passes through the fallback path rather than the main rule.
Related: #11595 (other TopPSampler bugs in the same run() method, fixed in #11596).
I'm happy to open a PR for either fix once the intended behaviour is confirmed.
To Reproduce
import logging
from haystack import Document
from haystack.components.samplers import TopPSampler
logging.basicConfig(level=logging.WARNING)
def docs_example():
return [
Document(content="Berlin", meta={"similarity_score": -10.6}),
Document(content="Belgrade", meta={"similarity_score": -8.9}),
Document(content="Sarajevo", meta={"similarity_score": -4.6}),
]
for top_p in [0.99, 0.95]:
sampler = TopPSampler(top_p=top_p, score_field="similarity_score")
kept = sampler.run(documents=docs_example())["documents"]
print(f"top_p={top_p}: kept {[d.content for d in kept]}")Output:
top_p=0.99: kept ['Sarajevo']
WARNING:haystack.components.samplers.top_p:Top-p sampling with p=0.95 resulted in no documents being selected. Returning the document with the highest score.
top_p=0.95: kept ['Sarajevo']
Softmax probabilities for this example, sorted by score:
| Document | Probability | Cumulative |
|---|---|---|
| Sarajevo | 0.9842 | 0.9842 |
| Belgrade | 0.0134 | 0.9976 |
| Berlin | 0.0024 | 1.0000 |
FAQ Check
- Have you had a look at our new FAQ page?
System:
- OS: Windows 11
- Haystack version: haystack-ai 3.5.0 (latest on PyPI;
top_p.pyonmainunchanged since fix: validate sampler min_top_k #12651) - Python: 3.12.10
- torch: 2.15.0+cpu
Activity
adityashankarv commentedon Nov 16, 2026
Thanks for the clear report. I think the fix is to keep the smallest prefix whose cumulative probability reaches
top_p(using>=), so the document that crosses the threshold is kept. That matches Holtzman et al. and the example in the docs. I have a patch with regression tests ready locally. @kai2055, happy to step back if you're already on it; otherwise I can open a PR once a maintainer confirms the intended behavior.kai2055 commentedon Nov 16, 2026
@adityashankarv thanks for looking into this! Since I reported it and offered to work on it, I'd like to take the PR myself. It would be my first contribution to Haystack. I'll wait for a maintainer to confirm which behaviour is intended, then open it with regression tests.
One point for the maintainers: changing the code would change which documents existing pipelines receive (on the docs example at
top_p=0.99, Belgrade would now be included too), so it may need an upgrade note. A docs-only fix wouldn't change behaviour.ptimizeroracle commentedon Nov 16, 2026
some numbers for the maintainers' call, on main (
top_p.pyunchanged since 4b4346b) with the real component: against the textbook nucleus rule (smallest prefix with cumulative probability >= p) it differs on 3650 of 6000 random score lists, always by exactly one document, the one that crossestop_p.test_run_score_fieldpins the current rule (scores 0.7, 0.01, 0.001 at 0.95 keep 2, the nucleus keeps 3), so a code fix also changes that test. the docstring example at 0.95 only passes through the empty-selection fallback (sarajevo alone is already 0.984).top_p=1.0keeps every document either way(ai-assisted, checked by me)