Skip to content

fix: poll job.status() once per job in Experiment.status() - #582

Merged
ko3n1g merged 2 commits into
NVIDIA-NeMo:mainfrom
andrewwhitecdw:bugfix/experiment-status-polls-once
Aug 7, 2026
Merged

fix: poll job.status() once per job in Experiment.status()#582
ko3n1g merged 2 commits into
NVIDIA-NeMo:mainfrom
andrewwhitecdw:bugfix/experiment-status-polls-once

Conversation

@andrewwhitecdw

Copy link
Copy Markdown
Contributor

Summary

Experiment.status() called job.status(runner=...) twice while building one status row: once for the display text and again for the returned dictionary. A backend failure therefore produced two identical ERROR tracebacks per status request and wasted a redundant status poll.

This was flagged as a non-blocking follow-up in #569.

Changes

  • Store job.status(runner=self._runner) in a local variable and reuse it for both the display string and the returned dict in nemo_run/run/experiment.py.
  • Add a regression test in test/run/test_experiment.py asserting job.status() is called exactly once for both return_dict=True and the default print path.

Testing

uv run -- pytest test/run/test_experiment.py::test_experiment_status_polls_job_once -xvs

Result: 1 passed.

Ruff lint and format also pass on the changed files.

Contributor guidelines

  • DCO sign-off included.
  • One fix per PR.
  • Includes regression test.

Experiment.status() called job.status(runner=...) twice while building
one status row: once for the display text and again for the returned
dictionary. A backend failure therefore produced two identical ERROR
tracebacks per status request and wasted a redundant status poll.

Store the status in a local variable and reuse it for both the display
string and the dict.

Add a regression test that asserts job.status() is called exactly once
for both return_dict=True and the default print path.

Signed-off-by: Andrew White <andrewh@cdw.com>
@ko3n1g

ko3n1g commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

/review

Copy link
Copy Markdown
Contributor

🔎 On it — resolving the pull request and reviewing it rigorously.

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as review.

Copy link
Copy Markdown
Contributor

💬 LGTM — #582

Verdict: No blocking findings.

Blocking findings: None.

Non-blocking findings: None.

Review details:
nemo_run/run/experiment.py:911-929 polls job.status() once and consistently reuses that result for both the rendered status and returned dictionary, eliminating the duplicate backend request without changing the API or fallback behavior.
test/run/test_experiment.py:659-680 covers both return_dict=True and the default display path, asserting exactly one status call with the experiment runner in each case.
• I reviewed both changed files, the full Experiment.status() implementation, Job.status()/JobGroup.status(), CLI and test callers, and current main. The intervening timestamp change on main is unrelated and produces no merge conflict.

Validation: All 18 non-skipped checks on the exact head f9e6067d2eee39a38363034c35686abec0afb6da pass, including the test matrix, lint, format, CodeQL, DCO, spell check, copyright, and secrets detection. git diff --check also passes. I could not independently run pytest because this sandbox has no Python runtime; CI provides the executable test coverage.

@ko3n1g
ko3n1g enabled auto-merge (squash) August 7, 2026 11:59
@ko3n1g
ko3n1g merged commit ca285bb into NVIDIA-NeMo:main Aug 7, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants