Issues

• make qt version work

  clienhancement
  quantiles-evals/quantiles#76 · Filter by author arschles opened on Aug 13, 2026

• Allow type = "custom_configuration" in addition to type = "custom_nocode"

  configuration
  quantiles-evals/quantiles#74 · Filter by author arschles opened on Aug 11, 2026

• Add a response_labels field to exact_match custom_nocode evals

  enhancementevaluation support
  quantiles-evals/quantiles#70 · Filter by author arschles opened on Aug 9, 2026

• Make benchmark registry benchmarks optionally contain qt version constraints

  enhancement
  quantiles-evals/quantiles#69 · Filter by author arschles opened on Aug 9, 2026

• Remove PubMedQA built-in benchmark code

  enhancement
  quantiles-evals/quantiles#64 · Filter by author arschles opened on Aug 3, 2026

• Add built-in benchmark for simpleqa-verified

  quantiles-evals/quantiles#60 · Filter by author arschles opened on Aug 2, 2026

• Add a label scoring style to custom_nocode benchmarks

  enhancementevaluation support
  quantiles-evals/quantiles#59 · Filter by author arschles opened on Aug 2, 2026

• Seperate custom no-code example by style

  documentationenhancement
  quantiles-evals/quantiles#53 · Filter by author phranzia opened on Jul 25, 2026

• Make sure CLI versions built/uploaded with the ci-release.yaml job have a correct qt --version

  CIenhancement
  quantiles-evals/quantiles#47 · Filter by author arschles opened on Jul 20, 2026

• Set up GitHub actions caching for CLI builds/tests/etc...

  CItesting
  quantiles-evals/quantiles#45 · Filter by author arschles opened on Jul 18, 2026

• Ensure all configuration, in --input flag and configuration file, to limit number of samples is named "samples"

  bug
  quantiles-evals/quantiles#42 · Filter by author arschles opened on Jul 15, 2026

• Add distribution metrics for input and output tokens

  enhancement
  quantiles-evals/quantiles#41 · Filter by author arschles opened on Jul 12, 2026

• Make the curl ... | bash run with sh instead

  enhancement
  quantiles-evals/quantiles#40 · Filter by author arschles opened on Jul 12, 2026

• Make qt CLI installable with homebrew

  enhancement
  quantiles-evals/quantiles#39 · Filter by author arschles opened on Jul 12, 2026

• Convert CLI tests to use cargo-nextest

  enhancementtesting
  quantiles-evals/quantiles#38 · Filter by author arschles opened on Jul 12, 2026

• Version the configuration file

  configurationenhancement
  quantiles-evals/quantiles#37 · Filter by author arschles opened on Jul 12, 2026

• Investigate and, where appropriate, remove #[expect(clippy::cast_precision_loss)] instances

  bug
  quantiles-evals/quantiles#36 · Filter by author arschles opened on Jul 12, 2026

• Add support for prompt templates right inside the toml file, rather than a separate file

  configuration
  quantiles-evals/quantiles#35 · Filter by author arschles opened on Jul 12, 2026

• Look for, and deduplicate, duplicated documentation

  documentation
  quantiles-evals/quantiles#34 · Filter by author arschles opened on Jul 12, 2026

• Create a newtype for run IDs

  enhancement
  quantiles-evals/quantiles#33 · Filter by author arschles opened on Jul 12, 2026

• Add more complete configuration-parsing tests

  testing
  quantiles-evals/quantiles#32 · Filter by author arschles opened on Jul 11, 2026

• Prefix huggingface datasets with hf:// or huggingface://

  enhancement
  quantiles-evals/quantiles#31 · Filter by author arschles opened on Jul 11, 2026

• Record the dataset that was used in a run

  enhancement
  quantiles-evals/quantiles#29 · Filter by author arschles opened on Jul 11, 2026

• Add support for running one benchmark over multiple models

  enhancement
  quantiles-evals/quantiles#28 · Filter by author arschles opened on Jul 11, 2026

• Add a qt delete command

  enhancement
  quantiles-evals/quantiles#26 · Filter by author arschles opened on Jul 11, 2026