Skip to content

ContextRelevanceEvaluator raises TypeError when relevant_statements is null #13757

Open

Description

@zhenqiangZ

Bug description

When the chat generator returns structurally valid JSON where relevant_statements is not a list (for example {"relevant_statements": null}, which JSON-mode models can emit when nothing relevant is found), ContextRelevanceEvaluator crashes with:

TypeError: object of type 'NoneType' has no len()

The error is raised in ContextRelevanceEvaluator._postprocess_results (len(res["relevant_statements"])), i.e. after _parse_reply, so the documented raise_on_failure=False behavior does not apply: the whole run fails instead of marking the affected row as an error and excluding it from the score.

A value of the wrong type is also silently accepted otherwise: {"relevant_statements": "Rome is the capital of Italy."} is marked "evaluated" with score 1, even though the output is documented as a list.

Reproduction

from haystack import component
from haystack.dataclasses import ChatMessage
from haystack.components.evaluators import ContextRelevanceEvaluator


@component
class NullStatementsGenerator:
    @component.output_types(replies=list[ChatMessage])
    def run(self, messages):
        return {"replies": [ChatMessage.from_assistant('{"relevant_statements": null}')]}


evaluator = ContextRelevanceEvaluator(
    chat_generator=NullStatementsGenerator(), raise_on_failure=False, progress_bar=False
)
print(
    evaluator.run(
        questions=["Who created the Python language?"],
        contexts=[["Python was created by Guido van Rossum."]],
    )
)

Actual: TypeError: object of type 'NoneType' has no len()

Expected with raise_on_failure=False: the row is reported as an error and the run completes:

{
    "results": [{"relevant_statements": [], "score": nan, "status": "error"}],
    "score": nan,
    "individual_scores": [nan],
}

Expected with raise_on_failure=True: a ValueError describing the invalid relevant_statements value.

Related

Version

haystack-ai 3.6.0-rc0

Activity

  1. zhenqiangZ commented on Nov 15, 2026

    @zhenqiangZ
    Author

    I opened #13758 with a fix: malformed relevant_statements values are now routed through the existing error-row path when raise_on_failure=False, and raise a ValueError when it is True. Feedback welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions