Skip to content

TopPSampler selects documents at or below top_p, but the docs describe exceeding it #13761

Open

Description

@kai2055

Describe the bug

The TopPSampler documentation and its implementation describe opposite rules for which documents are kept.

The TopPSampler docs page says:

The practical goal of the TopPSampler is to return a list of documents that, in sum, have a score larger than the top_p value.

But TopPSampler.run() in haystack/components/samplers/top_p.py selects documents with:

condition = (cumulative_probs <= top_p) | close_to_top_p

The docs describe keeping documents until their cumulative probability exceeds top_p. The code keeps documents only while their cumulative probability stays at or below top_p, so the document that would push the total above top_p is dropped.

On the example from the docs page, the two rules return different documents (see To Reproduce).

Error message

No error is raised. The component runs normally and silently returns a different set of documents than the documentation describes.

Expected behavior

Following the docs, TopPSampler(top_p=0.99) on the docs page example should return Sarajevo and Belgrade (cumulative probability 0.9976). The code returns only Sarajevo (0.9842).

It's unclear to me which behaviour is intended, so either:

  • the docs should describe the current rule (keep documents while their cumulative probability stays at or below top_p), or
  • the code should follow the documented rule.

For reference, standard top-p sampling (Holtzman et al., 2019) keeps the smallest set whose cumulative probability reaches p, which matches the docs rather than the code.

Additional context

The example in the TopPSampler docstring uses top_p=0.95 and expects only Sarajevo. Under the current rule, no document is selected at all, because Sarajevo alone already has a cumulative probability of 0.9842, above 0.95. Sarajevo is returned only through the fallback, which logs a warning (see output below). So the docstring example passes through the fallback path rather than the main rule.

Related: #11595 (other TopPSampler bugs in the same run() method, fixed in #11596).

I'm happy to open a PR for either fix once the intended behaviour is confirmed.

To Reproduce

import logging
from haystack import Document
from haystack.components.samplers import TopPSampler

logging.basicConfig(level=logging.WARNING)

def docs_example():
    return [
        Document(content="Berlin", meta={"similarity_score": -10.6}),
        Document(content="Belgrade", meta={"similarity_score": -8.9}),
        Document(content="Sarajevo", meta={"similarity_score": -4.6}),
    ]

for top_p in [0.99, 0.95]:
    sampler = TopPSampler(top_p=top_p, score_field="similarity_score")
    kept = sampler.run(documents=docs_example())["documents"]
    print(f"top_p={top_p}: kept {[d.content for d in kept]}")

Output:

top_p=0.99: kept ['Sarajevo']
WARNING:haystack.components.samplers.top_p:Top-p sampling with p=0.95 resulted in no documents being selected. Returning the document with the highest score.
top_p=0.95: kept ['Sarajevo']

Softmax probabilities for this example, sorted by score:

Document Probability Cumulative
Sarajevo 0.9842 0.9842
Belgrade 0.0134 0.9976
Berlin 0.0024 1.0000

FAQ Check

System:

Activity

  1. added theissue type on Nov 15, 2026
  2. added
    P2Medium priority, add to the next sprint if no P1 available
    on Nov 16, 2026
  3. adityashankarv commented on Nov 16, 2026

    @adityashankarv

    Thanks for the clear report. I think the fix is to keep the smallest prefix whose cumulative probability reaches top_p (using >=), so the document that crosses the threshold is kept. That matches Holtzman et al. and the example in the docs. I have a patch with regression tests ready locally. @kai2055, happy to step back if you're already on it; otherwise I can open a PR once a maintainer confirms the intended behavior.

  4. kai2055 commented on Nov 16, 2026

    @kai2055
    Author

    @adityashankarv thanks for looking into this! Since I reported it and offered to work on it, I'd like to take the PR myself. It would be my first contribution to Haystack. I'll wait for a maintainer to confirm which behaviour is intended, then open it with regression tests.

    One point for the maintainers: changing the code would change which documents existing pipelines receive (on the docs example at top_p=0.99, Belgrade would now be included too), so it may need an upgrade note. A docs-only fix wouldn't change behaviour.

  5. ptimizeroracle commented on Nov 16, 2026

    @ptimizeroracle

    some numbers for the maintainers' call, on main (top_p.py unchanged since 4b4346b) with the real component: against the textbook nucleus rule (smallest prefix with cumulative probability >= p) it differs on 3650 of 6000 random score lists, always by exactly one document, the one that crosses top_p. test_run_score_field pins the current rule (scores 0.7, 0.01, 0.001 at 0.95 keep 2, the nucleus keeps 3), so a code fix also changes that test. the docstring example at 0.95 only passes through the empty-selection fallback (sarajevo alone is already 0.984). top_p=1.0 keeps every document either way

    (ai-assisted, checked by me)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority, add to the next sprint if no P1 available

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions