Issues

• Proposal: Add BFSI hazard domain — FinProof Bench (Banking/Financial Services/Insurance)

  mlcommons/ailuminate#38 · Filter by author msrkrishna-ship-it opened on Sep 13, 2026
  1

• Hazard categories are scored on text responses. Physical-AI systems fail in the action space.

  mlcommons/ailuminate#37 · Filter by author sattyamjjain opened on Sep 1, 2026

• Proposal: vulnerable-user reliability overlay for AILuminate (Korean pilot)

  mlcommons/ailuminate#36 · Filter by author vivwon3737 opened on Aug 4, 2026

• Refactor top-level information architecture

  mlcommons/ailuminate#30 · Filter by author mikek-mlcommons opened on Apr 28, 2025

• Will test sets in other languages/regions be open source?

  mlcommons/ailuminate#28 · Filter by author 486zzx opened on Apr 11, 2025
  1

• Undocumented hazard category: spc_lgl

  mlcommons/ailuminate#26 · Filter by author paulinek13 opened on Mar 25, 2025

• How do we obtain evaluation metrics for this demo subset?

  mlcommons/ailuminate#23 · Filter by author ljvmiranda921 opened on Mar 14, 2025
  2

• Benchmark the latest high visibility models

  mlcommons/ailuminate#21 · Filter by author mikek-mlcommons opened on Mar 10, 2025
  1

• Benchmark is accessible by the users via an API and integrated onto hyperscale or other 3P platforms (e.g., Hugging Face)

  mlcommons/ailuminate#20 · Filter by author mikek-mlcommons opened on Mar 6, 2025
  

• Rename ModelBench to AILuminate

  mlcommons/ailuminate#18 · Filter by author mikek-mlcommons opened on Feb 27, 2025
  

• Test and publish latest "model of the moment" results: DeepSeek, etc

  mlcommons/ailuminate#17 · Filter by author mikek-mlcommons opened on Feb 27, 2025
  

• Mike documents use cases

  mlcommons/ailuminate#15 · Filter by author mikek-mlcommons opened on Feb 24, 2025
  

• Make it clearer how people can participate

  mlcommons/ailuminate#11 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  2

  

• Make it easier to run A/B tests with SUTs

  mlcommons/ailuminate#10 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  1

  

• Get per-prompt scores from DEMO set

  mlcommons/ailuminate#9 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  1

  

• Make it easier to get access to the official evaluator

  mlcommons/ailuminate#8 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  

• SUT version/time recording (“GPT-4o at 8:53AM Jan 17, 2025) etc.

  mlcommons/ailuminate#7 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  

• Easier onboarding

  mlcommons/ailuminate#6 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  2

  

• Make it easier to do offline asynchronous tests

  mlcommons/ailuminate#5 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  1

  

• Review current docs with Kurt and William

  mlcommons/ailuminate#4 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  1

  

• Clean up ModelBench documentation

  mlcommons/ailuminate#3 · Filter by author mikek-mlcommons opened on Feb 11, 2025
  1

  

• Make it easier to do offline benchmarking

  mlcommons/ailuminate#2 · Filter by author mikek-mlcommons opened on Jan 16, 2025
  1