Repository navigation
[codex] publish 26.07 release notes, migration checklist, and cut the 26.07 GA docs train - #2143
Conversation
Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
78a34db to
cbb22af
Compare
Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
Greptile SummaryThis PR publishes the NeMo Curator 26.07 (v1.3.0) documentation rollup: it replaces the stale 26.04 release notes page with categorized 26.07 notes and adds a new step-by-step migration checklist. All other touched pages receive targeted corrections to Python version ranges and stack-version references that previously pointed to 26.04/26.06 language.
Confidence Score: 5/5Documentation-only change; no runtime code is modified and all internal cross-links use the correct numeric-prefixed anchor format. Every changed file is pure documentation (MDX, YAML nav config). Python version ranges, stack version references, and forward-looking deprecation warnings are all updated consistently across the seven files. The new migration checklist anchors in index.mdx match the heading slugs generated for the numbered sections in 26-07-migration.mdx. The previous review concern about numeric-prefixed anchors has been addressed — all fragment links now include the step-number prefix. Files Needing Attention: No files require special attention. The migration checklist and release notes are internally consistent and cross-linked correctly. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Release Notes Navigation] --> B[Overview - index.mdx: 26.07 release notes]
A --> C[26.07 Migration Checklist - 26-07-migration.mdx: NEW]
A --> D[Migration FAQ - migration-faq.mdx]
B -->|links to| C
C -->|links back to| B
B --> E[Breaking Changes section]
E --> C
C --> F[Step 1: Upgrade Python]
C --> G[Step 2: Stop Supplying task_id]
C --> H[Step 3: Instantiate Sentinel Tasks]
C --> I[Step 4: Remove Video Preprocessing Option]
C --> J[Step 5: Replace Unversioned Caption Variant]
C --> K[Step 6: Replace InferenceModelConfig]
C --> L[Step 7: Separate ASR DataLoader Workers]
C --> M[Step 8: Review Architecture Dependencies]
C --> N[Step 9: Configure Resumability]
C --> O[Step 10: Run Post-Upgrade Checks]
style C fill:#d4edda,stroke:#28a745
style B fill:#d1ecf1,stroke:#17a2b8
Reviews (8): Last reviewed commit: "Merge branch 'main' into codex/docs-26-0..." | Re-trigger Greptile |
Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
303c2a2 to
28712ec
Compare
Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
|
Adversarial release-doc review follow-up is now included in commit
This makes #2147 and #2148 explicit content dependencies of the release rollup. Please merge the detailed feature PRs first and keep #2143 last so the consolidated 26.06 page can be rebased once and retain all release-note material. Validation after rebasing on the current remote branch:
Cross-PR integration and publication staging are tracked in #2160. |
|
@meatybobby friendly ping on this docs PR for the #2118 release-docs workstream — when you have a moment, could you take a look at the 26.06 release notes and migration checklist? Docs preview: https://nvidia-preview-codex-docs-26-06-release-notes.docs.buildwithfern.com/nemo/curator Thanks! |
lbliii
added
the
docs-release-26.07
ChatGPT Codex Connector
|
@sarahyurick, thanks for already taking a look here. Once your comments are addressed, could you please provide a formal approval if the 26.06 release notes and migration checklist look ready? Thank you! |
|
|
||
| ```python | ||
| @dataclass | ||
| class MyGpuStage(ProcessingStage[AudioTask, AudioTask]): |
There was a problem hiding this comment.
It doesn't declare GPU resources.
There was a problem hiding this comment.
Fixed — the example was labelled a GPU stage but declared no resources. Added, along with the imports the snippet needs:
resources: Resources = field(
default_factory=lambda: Resources(gpu_memory_gb=16.0)
)Used gpu_memory_gb rather than gpus since this is a single-GPU stage, per the guidance on the Resources page.
| filtered = df[df["score"] > self.threshold] | ||
|
|
||
| if filtered.empty: | ||
| return None # Filter out entire batch |
There was a problem hiding this comment.
Should return NoneTask now, since it is supposed to represent a filtered task.
But I guess it is okay if we do not update the documentation for this yet.
There was a problem hiding this comment.
Added a short note rather than changing the example, since returning None is still the supported way to filter a batch out:
Returning
Noneremains the supported way to filter a batch out. The resumability layer converts it to aNoneTask, which represents a filtered task and consumes that branch, so aNonereturn is what you will see surfaced asNoneTaskdownstream.
Happy to go further and rewrite the snippet to construct NoneTask directly if you would rather document it that way — just say the word.
| """ | ||
| ``` | ||
|
|
||
| ### `build()` |
There was a problem hiding this comment.
Users shouldn't ever need to interact with build I think.
There was a problem hiding this comment.
Agreed. Left the section in place since it explains what happens between add_stage() and execution, but flagged it as not user-facing:
run()callsbuild()for you. You should not need to call it directly; it is documented here to explain what happens betweenadd_stage()and execution.
| ray_stage_spec={ | ||
| RayStageSpecKeys.RAY_NUM_CPUS: 1.0, | ||
| }, | ||
| xenna_stage_spec={ | ||
| "worker_max_lifetime_m": 45, | ||
| }, |
There was a problem hiding this comment.
Probably folks would never specify both ray_stage_spec and xenna_stage_spec right? Just whichever executor they are using for the pipeline?
There was a problem hiding this comment.
Right — reworked the example so each spec is paired with the executor it belongs to, instead of showing both set at once. The main snippet now sets only ray_stage_spec, followed by a Xenna equivalent, with:
Both specs are shown together here only to document the arguments. In practice you set the one matching the executor you run the pipeline with [...] Setting both is harmless — each executor reads only its own spec — but it is only useful if the same stage runs under both backends.
Let me know if the last sentence overstates it and you would rather I just drop it.
Applies @sarahyurick's review. Each change is mirrored in fern/versions/main/pages and the frozen fern/versions/v26.07/pages so the two trees stay byte-identical. Corrections verified against source: - Fix the SLURM client name. The class is SlurmRayClient, not RaySlurmClient; the documented import would have raised ImportError. Also note that it subclasses RayClient and behaves identically on a single node or outside SLURM. - Fix the RayActorPoolExecutor import. nemo_curator.backends.experimental no longer exists; the executor lives in nemo_curator.backends.ray_actor_pool. - Fix the NeMo Data Designer install. data-designer ships in sdg_cpu and sdg_cuda12, not the text extras. - Drop Dask from the container component list. It is present only as a transitive dependency of the RAPIDS raft-dask-cu12 package, not as an execution framework. - Declare GPU resources on the audio batch-stage example, with the imports the snippet needs. - Note that Pipeline.build() is called by run() and is not part of the user-facing flow. - Split the with_() example so each stage spec is paired with its own executor instead of setting both at once. - Note that a None return surfaces downstream as NoneTask. - Drop 'and others' from the streaming executor list. Removals: - Delete the SLURM image-curation page. It used ImageTextPairDataset and nemo_curator.utils.distributed_utils.get_client, both removed with Dask. - Delete the experimental-executors API page, whose module no longer exists. - Remove both nav entries, the incoming cards and cross-references, and add redirects for the unversioned, latest, main, and v26.07 URLs. Older trains still host both pages and are left alone. Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
|
Thanks @sarahyurick — all 11 comments addressed in Three of your comments turned out to be broken code rather than just stale prose, so flagging those specifically:
Both deletions confirmed against source before removing: the SLURM image page used Two I answered rather than changed outright — Validation: /ok to test 3fc7eea |
The Data Designer edit added two lines above the placeholder api_key="unused" example, shifting it from line 183 to 185 in both the main and v26.07 trees. detect-secrets pins line numbers, so the baseline needed realigning. Same hash, no new secret material. Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com>
|
/ok to test b37123bd0e0dfd57ad2eb6a4e2fbb7b1ac2e8ac1 |
|
/ok to test b37123b |
@lbliii, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
|
||
| ```bash | ||
| uv pip install --extra-index-url https://pypi.nvidia.com nemo-curator[text_cuda12] | ||
| uv pip install nemo-curator[sdg_cpu] |
There was a problem hiding this comment.
Sorry for the confusion, I meant I think the NDD stages need both the text and SDG extras, not just one or the other.
(And the inference server extra as you already included below.)
There was a problem hiding this comment.
@lbliii Can you please add this in a followup? Thanks. I will merge the PR now to ensure no rebasing is required later.
This branch was previously deployed
4 participants
Summary
Publishes the NeMo Curator 26.07 (v1.3.0) release-level documentation rollup tracked by #2120 and the supplemental code-freeze audits in #2118.
maindocs train with categorized 26.07 release notes.Review feedback addressed
Per @ayushdg's review:
26-07-migration), nav entry, section anchors, and every cross-link now say 26.07. The file was renamed26-06-migration.mdx->26-07-migration.mdx.uv venv --python 3.11. Step 1 now leads with theuvflow and keeps the stdlibvenvcommands as the non-uvalternative.with_()and stage-spec standardization from #2077 / #2086 stays documented as a feature under "Consistent Stage Scheduling andwith_()Overrides" in the release notes.Also refreshed the stale forward-looking notices on the
maintrain that still said "Python 3.10 support will be removed in 26.06" and listed Python 3.10-3.12. They now state that 3.10 support was removed in 26.07 and list 3.11-3.13, matchingrequires-python = ">=3.11,<3.14". Bumped the two "tested 26.06 stack" references in the inference-server guide.@sarahyurick — the dependency table still matches
r1.3.0'spyproject.toml; leaving the request for @ayushdg / @praateekmahajan / @suiyoubi to confirm whether any entry needs extra context.26.07 version cut (GA)
This PR now also freezes the
mainpreview tree as the 26.07 GA train and repointslatest, so publishing after merge makes 26.07 GA:fern/versions/main/pages/->fern/versions/v26.07/pages/(177 pages, verified byte-identical tomain).fern/versions/v26.07.ymlwith./main/path prefixes rewritten to./v26.07/(nav otherwise identical tomain.yml).versions/latest.ymlsymlink fromv26.04.ymltov26.07.yml, so/latest/...serves 26.07.26.07 · v1.3.0entry todocs.ymlversions:and relabels Latest asLatest · v1.3.0 (26.07), matchingnemo_curator/package_info.py.index.html, calendar-train26.07paths, nested:path*/index.html, Sphinx-style:path*.html), each placed to match the existing per-version ordering ahead of the catch-alls.mainstays as theMain · previewtrain and keeps moving forward;v26.04,v26.02, andv25.09are untouched.Note that merging alone does not publish. Production updates only on a
docs/v*tag push (or manual dispatch ofpublish-fern-docs.yml).Two follow-ups intentionally left out of scope:
fern/README.mdsays to "demote/remove the oldest GA per the support policy" — I do not know the policy, so all four trains remain listed.CHANGELOG.mdhas no1.3.0section (it stops at1.0.0).fern/README.mdasks that version display-names align withCHANGELOG.md; the display-name matchespackage_info.py, but the changelog itself needs a separate update.Release-boundary update
PR #2063 is now on
main, so the branch no longer says resumability is unreleased. The release notes and migration checklist document releasedPipeline.run(checkpoint_path=...)behavior, its pre-existing-Ray requirement, local checkpoint storage, resumability-safe stage validation, andNoneTask/FailedTasksemantics.PR #2059 is also included as a new capability: deterministic source sharding across SLURM arrays, shard completion manifests, and incomplete-shard retry discovery.
Supplemental coverage
The update explicitly captures #1427, #1679, #1746, #1762, #1763, #1774, #1844, #1855, #1870, #1880, #1888, #1890, #1895, #1927, #1957, #2059, and #2063. Detailed #1927 guidance is in #2149, native resumability is in #2134, and SLURM arrays are in #2154.
Merge order / documentation dependencies
Merge this PR after the detailed feature workstreams so its canonical internal links resolve to their final pages:
Validation
main(merge conflict inindex.mdxresolved in favor of the 26.07 rewrite; the conflicting hunk was the 26.04 InferenceServer section that this PR removes).fern check: 0 errorsgit diff --check: passedast26-07-migration#...link resolves to a heading in the renamed file26.06/26-06references underfern/versions/main/fern checkre-run after the version cut: 0 errorspath:entries inv26.07.ymlresolve to real files;latest.ymlresolves tov26.07.ymldiff -rconfirmsversions/v26.07is byte-identical toversions/main, and the two nav files are identical modulo the path prefixmainCloses #2120
Part of #2144
Parent tracker: #2118