Skip to content

evaluate_rubrics.py fails to parse valid JSON when string values contain Markdown code fences #15

Description

@Refrain-NJU

Question Description

The LLM response parsing logic in rf/task_id/tests/evaluate_rubrics.py does not fully account for Markdown code fences appearing inside JSON string values.

The parser currently uses logic like:

    if "```json" in text:
        after = text[text.find("```json") + 7:]
        end = after.find("```")
        if end != -1:
            text = after[:end].strip()

This assumes that the first closing-code-fence sequence after the outer ```json marker terminates the JSON block. However, an LLM may include a fenced code example inside a JSON string, especially in a justification field.
For example:

```json
{
  "ratings": [
    {
      "status": "YES",
      "justification": "The diff shows that a new `CannotSetMailbox` exception class is created in `app/user_settings.py` with a `msg` attribute, following the exact same pattern as the existing `CannotSetAlias` class. The code added is:\n\n```python\nclass CannotSetMailbox(Exception):\n    def **init**(self, msg: str):\n        self.msg = msg\n```\n\nThis mirrors the existing `CannotSetAlias` class structure which has the same `**init**` method signature and `msg` attribute assignment."
    }
  ]
}```

In this case, after.find("```") finds the ```python fence inside the justification value instead of the closing fence surrounding the JSON response. The extracted JSON is therefore truncated and cannot be parsed.

Impact

The JSON parsing fails even though the LLM response contains a usable ratings object. As a result, the response may be skipped and the evaluation is not performed.

Suggested fix

The simplest fix is to use end = after.rfind("```") so that the outermost closing code fence is used to extract the complete response.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions