You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ContextRelevanceEvaluator raises TypeError when relevant_statements is null #13757
When the chat generator returns structurally valid JSON where relevant_statements is not a list (for example {"relevant_statements": null}, which JSON-mode models can emit when nothing relevant is found), ContextRelevanceEvaluator crashes with:
TypeError: object of type 'NoneType' has no len()
The error is raised in ContextRelevanceEvaluator._postprocess_results (len(res["relevant_statements"])), i.e. after _parse_reply, so the documented raise_on_failure=False behavior does not apply: the whole run fails instead of marking the affected row as an error and excluding it from the score.
A value of the wrong type is also silently accepted otherwise: {"relevant_statements": "Rome is the capital of Italy."} is marked "evaluated" with score 1, even though the output is documented as a list.
Reproduction
fromhaystackimportcomponentfromhaystack.dataclassesimportChatMessagefromhaystack.components.evaluatorsimportContextRelevanceEvaluator@componentclassNullStatementsGenerator:
@component.output_types(replies=list[ChatMessage])defrun(self, messages):
return {"replies": [ChatMessage.from_assistant('{"relevant_statements": null}')]}
evaluator=ContextRelevanceEvaluator(
chat_generator=NullStatementsGenerator(), raise_on_failure=False, progress_bar=False
)
print(
evaluator.run(
questions=["Who created the Python language?"],
contexts=[["Python was created by Guido van Rossum."]],
)
)
Actual: TypeError: object of type 'NoneType' has no len()
Expected with raise_on_failure=False: the row is reported as an error and the run completes:
I opened #13758 with a fix: malformed relevant_statements values are now routed through the existing error-row path when raise_on_failure=False, and raise a ValueError when it is True. Feedback welcome.
Activity
zhenqiangZ commentedon Nov 15, 2026
I opened #13758 with a fix: malformed
relevant_statementsvalues are now routed through the existing error-row path whenraise_on_failure=False, and raise aValueErrorwhen it isTrue. Feedback welcome.