Repository navigation
Reconcile training resume and reject incomplete RNG state - #7
Conversation
--save_state_every_steps N writes checkpoints/step-* every N optimizer steps: the usual servable checkpoint plus trainer_state.pt (optimizer, scheduler, epoch and micro-batch position, counters, early-stopping state) and per-rank torch/CUDA RNG. --resume_from continues from one of them into a new --out. The data order is replayed from the seed and augmentation is already seeded per record, so a resumed run reproduces the uninterrupted weights bit for bit (tested for SFT and RLCR from every intermediate step, including epoch boundaries). Trajectory-changing settings and the process count must match; mismatches are rejected before training. Both flags default to off, leaving existing runs and outputs unchanged.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Human Review
Human Review: see the findings and review evidence. AI semantic review has not completed.
Reviewed head |
|
This conflicts with current |
Keep the current resume interface and checkpoint validation. Adapt the original contribution's cadence and per-step SFT/RLCR parity coverage, and reject incomplete per-rank RNG state before creating the output. Co-authored-by: Xuying Ning <xuying@meta.com>
ZhiningLiu1998
changed the title
ZhiningLiu1998
left a comment
ZhiningLiu1998
left a comment
There was a problem hiding this comment.
Reconciled the duplicate resume implementation with current main and preserved the original per-step parity coverage. All 51 focused checks pass, including SFT/RLCR recovery and early rejection of incomplete RNG state; both full CPU CI jobs and site checks pass.
2 participants
Reconcile the original resume contribution with the implementation now on main, retaining
--resumeand--checkpoint-every-steps, checkpoint fingerprints, schedule validation, and prior-best/early-stop state. The original commits remain in this branch's history.Port the contributor's SFT/RLCR checks for checkpoint cadence and recovery from every unfinished optimizer-step boundary. Incomplete per-rank RNG state now raises an actionable error before an output directory is created, rather than failing later with a missing-key, index, or attribute error.
Validation: 51 focused training/resume checks passed; 2 tests requiring an external trained checkpoint were skipped. Both SFT and RLCR match continuous-training tensors and counters after resuming from steps 1 through 5, and finished schedules are rejected.