Issues

• Run e2e test less often

  chore
  mlcommons/modelbench#1686 · Filter by author rogthefrog opened on Sep 29, 2026
  

• Fix dynamic openai suts

  mlcommons/modelbench#1682 · Filter by author bkorycki opened on Sep 25, 2026

• run modelbench jobs off the modelrunner machine (cloud run)

  chore
  mlcommons/modelbench#1680 · Filter by author rogthefrog opened on Sep 25, 2026
  

• figure out huggingface billing and endpoint health

  mlcommons/modelbench#1679 · Filter by author rogthefrog opened on Sep 25, 2026
  

• Improve benchmark object design

  chore
  mlcommons/modelbench#1676 · Filter by author bkorycki opened on Sep 23, 2026

• Prism collects Candela run data

  mlcommons/modelbench#1674 · Filter by author bollacker opened on Sep 23, 2026
  2

  

• Support benchmark SKUs/UIDs for initialization in Candela

  mlcommons/modelbench#1673 · Filter by author bollacker opened on Sep 23, 2026
  2

  

• Stand up security 1.1 benchmark.

  mlcommons/modelbench#1672 · Filter by author bollacker opened on Sep 23, 2026

• Optional EvalPort interop for HazardScore/BenchmarkScore results

  mlcommons/modelbench#1669 · Filter by author adhabnr-ux opened on Sep 22, 2026

• Why can't we run any NVIDIA NIM suts

  mlcommons/modelbench#1660 · Filter by author bkorycki opened on Sep 14, 2026

• Fix mistral SUT model id formatting

  mlcommons/modelbench#1659 · Filter by author bkorycki opened on Sep 14, 2026
  1

• Misc google SUT failures

  mlcommons/modelbench#1658 · Filter by author bkorycki opened on Sep 14, 2026
  1

• Some Google SUTs don't return real content

  mlcommons/modelbench#1657 · Filter by author bkorycki opened on Sep 14, 2026
  2

• Update William's tool to only check models we care about

  mlcommons/modelbench#1656 · Filter by author bkorycki opened on Sep 14, 2026

• Decide if we should support Google's Interactions API

  mlcommons/modelbench#1655 · Filter by author bkorycki opened on Sep 14, 2026

• Fix AWS SUTs

  mlcommons/modelbench#1654 · Filter by author bkorycki opened on Sep 14, 2026

• Anthropic SUT sometimes doesn't return a TextBlock

  mlcommons/modelbench#1653 · Filter by author bkorycki opened on Sep 14, 2026
  1

• Create openrouter adapter

  mlcommons/modelbench#1648 · Filter by author bollacker opened on Sep 11, 2026
  1

• Add default model lists to dynamic SUT providers that don't auto-discover

  mlcommons/modelbench#1646 · Filter by author wpietri opened on Sep 9, 2026

• Investigate dynamic SUT run failures

  mlcommons/modelbench#1645 · Filter by author wpietri opened on Sep 9, 2026

• Investigate dynamic SUT listing failures

  mlcommons/modelbench#1644 · Filter by author wpietri opened on Sep 9, 2026
  1

• consistency checker handles new scoring

  mlcommons/modelbench#1641 · Filter by author bkorycki opened on Sep 3, 2026

• Candela server and client support safety benchmark versions

  mlcommons/modelbench#1640 · Filter by author rogthefrog opened on Sep 3, 2026

• Confirm that all results from production run are in metadata

  mlcommons/modelbench#1638 · Filter by author bollacker opened on Sep 2, 2026

• Integrity check modelplane vs production for a couple of riki's SUTs

  mlcommons/modelbench#1637 · Filter by author bollacker opened on Sep 2, 2026