Repository navigation

[benchmarking] Multiple fixes to stabilize the nightly benchmark suite - #2035

Merged

rlratzel merged 20 commits into

NVIDIA-NeMo:mainfrom

rlratzel:2606-update_benchmark_env_check

Jun 4, 2026

Commits on May 29, 2026
- [benchmarking] Make undefined env vars in config non-fatal by default
  
  rlratzel
  and
  claude
  committed
  b737202
- benchmark: bump ndd_ray_serve_dp4 timeout_s 700 -> 1200
  
  rlratzel
  and
  claude
  committed
  ab1d438
- benchmark: add --entry-exact-name flag for exact entry matching
  
  rlratzel
  and
  claude
  committed
  e8c6a52
- benchmark: rename --entry-exact-name to --entries-exact (list, strict)
  
  rlratzel
  and
  claude
  committed
  7b35169
- benchmark: bump exact_dedup_identification timeout_s 500 -> 1500
  
  rlratzel
  and
  claude
  committed
  21647b4
Commits on Jun 1, 2026
- benchmark: install lynx in ci_benchmark_launcher.sh
  
  rlratzel
  and
  claude
  committed
  d0eac90
- benchmark: preserve scratch dir for fuzzy_dedup_identification
  
  rlratzel
  and
  claude
  committed
  6369150
- benchmark: range domain_label_games_count +/-5% (was exact)
  
  rlratzel
  and
  claude
  committed
  0c375b7
Commits on Jun 2, 2026
- Merge remote-tracking branch 'upstream/main' into 2606-update_benchmark_env_check
  rlratzel
  committed
  3759b21
- benchmark: bump dedup_removal_* timeout_s for real-data wall
  
  rlratzel
  and
  claude
  committed
  c2f3a6b
- Revert "benchmark: preserve scratch dir for fuzzy_dedup_identification"
  
  rlratzel
  and
  claude
  committed
  a3c38ac
- Add optional /dev/shm cap via CURATOR_SHM_SIZE_BYTES env var
  
  rlratzel
  and
  claude
  committed
  7cf6523
- Add optional all-GPU utilization poller via CURATOR_GPU_POLL_INTERVAL_S
  
  rlratzel
  and
  claude
  committed
  78c5afd
- Raise A100 container memory cap to 2 TiB; drop EOS-side shm remount
  
  rlratzel
  and
  claude
  committed
  fd6e9cf
Commits on Jun 3, 2026
- Replace bash GPU poller with threaded GPUStatsRecorder
  
  rlratzel
  and
  claude
  committed
  c2fd47e
- Wire NEMO_CI_* env vars + Slack viewer-URL block (curator-side)
  
  rlratzel
  and
  claude
  committed
  da4c676
Commits on Jun 4, 2026
- GPUStatsRecorder: also record power draw, power limit, fan speed
  
  rlratzel
  and
  claude
  committed
  40f8d4f
- Tighten domain_label_games_count tolerance from +/-5% to +/-1%
  
  rlratzel
  and
  claude
  committed
  ab31cf1
- nightly-benchmark.yaml: drop noisy timeout_s commentary
  
  rlratzel
  and
  claude
  committed
  c580964
- Merge remote-tracking branch 'upstream/main' into 2606-update_benchmark_env_check
  rlratzel
  committed
  c15025d

Navigation Menu

Repository navigation

[benchmarking] Multiple fixes to stabilize the nightly benchmark suite - #2035

[benchmarking] Multiple fixes to stabilize the nightly benchmark suite#2035
rlratzel merged 20 commits into
NVIDIA-NeMo:mainfrom
rlratzel:2606-update_benchmark_env_check

Commits on May 29, 2026

[benchmarking] Make undefined env vars in config non-fatal by default

benchmark: bump ndd_ray_serve_dp4 timeout_s 700 -> 1200

benchmark: add --entry-exact-name flag for exact entry matching

benchmark: rename --entry-exact-name to --entries-exact (list, strict)

benchmark: bump exact_dedup_identification timeout_s 500 -> 1500

Commits on Jun 1, 2026

benchmark: install lynx in ci_benchmark_launcher.sh

benchmark: preserve scratch dir for fuzzy_dedup_identification

benchmark: range domain_label_games_count +/-5% (was exact)

Commits on Jun 2, 2026

Merge remote-tracking branch 'upstream/main' into 2606-update_benchmark_env_check

benchmark: bump dedup_removal_* timeout_s for real-data wall

Revert "benchmark: preserve scratch dir for fuzzy_dedup_identification"

Add optional /dev/shm cap via CURATOR_SHM_SIZE_BYTES env var

Add optional all-GPU utilization poller via CURATOR_GPU_POLL_INTERVAL_S

Raise A100 container memory cap to 2 TiB; drop EOS-side shm remount

Commits on Jun 3, 2026

Replace bash GPU poller with threaded GPUStatsRecorder

Wire NEMO_CI_* env vars + Slack viewer-URL block (curator-side)

Commits on Jun 4, 2026

GPUStatsRecorder: also record power draw, power limit, fan speed

Tighten domain_label_games_count tolerance from +/-5% to +/-1%

nightly-benchmark.yaml: drop noisy timeout_s commentary

Merge remote-tracking branch 'upstream/main' into 2606-update_benchmark_env_check