[codex] docs: add Nemotron OCR pipeline guide

[lbliii]

Summary

• add a Nemotron OCR concept page covering architecture, installation, input sources, schemas, stage defaults, verifier behavior, and failure/restart semantics
• add a runnable tutorial for OCR-only and optional NVIDIA Inference API scoring/QA paths
• document the six user-visible QA interaction shapes, four internal QA tags, and all 11 dense-answer formats
• provide CLI and Python examples, complete argument reference, result schemas, batching behavior, secrets handling, and shard lifecycle guidance
• add navigation and cross-links from synthetic data, image curation, image tutorials, and LLM client documentation

Current API boundary

Issue #2121 names several APIs that were not present in the final merged #1899 implementation. These docs use the actual current names and behavior:

• OCRNemotronV2Stage, not NemotronOCRV2Stage
• HFDatasetImageReaderStage and JsonlSampleWriterStage, not HFDatasetImageReader / ResultWriterStage
• NVInferenceClient, not NVInferenceModel
• OCRDenseItem, not OCRDenseWord
• conversation and dense-QA helper functions called by OCRScoringQAStage; there are no standalone OCRConversationalizeStage or OCRDenseQAStage classes
• no OCR-specific TarImageReader, ParquetReader, or SkipProcessedStage

The pages call this out directly and document custom ImageSampleTask[OCRData] input as the extension point instead of publishing non-runnable configuration.

Validation

• fern check — 0 errors (103 existing warnings)
• fern docs broken-links — no errors in changed pages; 22 pre-existing API-reference errors elsewhere
• documented scoring CLI arguments build the expected four-stage pipeline
• focused OCR, task, conversation, model-stage, and NVIDIA client tests — 120 passed, 1 live GPU model-load test deselected
• python -m py_compile for the tutorial and all documented source modules
• git diff --cached --check

The repository's Linux-only import guard was bypassed for local CPU unit validation on macOS. The live NemotronOCR-v2 model load requires the documented container and GPU and was not run locally.

Closes #2121

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR adds a new documentation section for the Nemotron OCR synthetic-data pipeline: a concept page covering architecture, schemas, verification settings, QA shapes, and failure semantics, plus a runnable tutorial with CLI and Python API examples. It also adds navigation entries and cross-links from the image curation, synthetic-data, and LLM client pages.

• The concept page accurately reflects the merged API (OCRNemotronV2Stage, OCRScoringQAStage, HFDatasetImageReaderStage, JsonlSampleWriterStage) — all import paths were verified against the actual source files.
• All CLI argument defaults in the tutorial reference table match the argparse definitions in tutorials/synthetic/omni/ocr_pipeline.py; the valid_only default divergence between JsonlSampleWriterStage and the CLI is called out explicitly.
• The conversation schema (nested {"conversation": {"conversation": [...]}} wrapper) is consistent between the concept page and tutorial step 5, addressing a prior review comment.

Confidence Score: 5/5

Documentation-only change; no executable logic is modified.

All documented API names, import paths, CLI argument names, and default values were verified against the actual source files. Previous reviewer comments on merge_output_shards placement, the JsonlSampleWriterStage valid_only default, and the conversation schema shape have all been addressed. No cross-page inconsistencies found.

No files require special attention.

Important Files Changed

Filename	Overview
fern/versions/main/pages/curate-text/synthetic/nemotron-ocr/index.mdx	New concept page: architecture table, installation, input sources, OCR data model, verifier settings, six QA shapes, eleven dense-answer formats, and conversation schema — all verified against actual source. Coordinate-ordering warning and nested conversation wrapper accurately documented.
fern/versions/main/pages/curate-text/synthetic/nemotron-ocr/tutorial.mdx	New tutorial: CLI and Python API examples verified against ocr_pipeline.py; all argument defaults match argparse definitions; merge_output_shards correctly placed in finally block; valid_only default divergence explicitly called out in prose.
fern/versions/main.yml	Adds Nemotron OCR section under synthetic data navigation with overview (slug: "") and tutorial entries; follows existing navigation conventions.
fern/versions/main/pages/curate-images/index.mdx	Adds a cross-link card pointing to the new Nemotron OCR page; link target and description are accurate.
fern/versions/main/pages/curate-images/tutorials/index.mdx	Adds a cross-link card to the OCR tutorial page; href and description match the new tutorial content.
fern/versions/main/pages/curate-text/synthetic/index.mdx	Adds a card entry for Nemotron OCR in the synthetic-data index with accurate description and tags.
fern/versions/main/pages/curate-text/synthetic/llm-client.mdx	Adds a Next Steps link to the Nemotron OCR page from the LLM client guide; link and description are accurate.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["HFDatasetImageReaderStage\n(Driver — 1 CPU)"] --> B["OCRNemotronV2Stage\n(Worker — 8 CPU + 1 GPU, batch 32)"]
    B --> C{"run_scoring_qa?"}
    C -->|No| D["JsonlSampleWriterStage\n(Worker — 2 CPU)"]
    C -->|Yes| E["OCRScoringQAStage\n(Worker — 1 CPU, batch 16)\n+ NVInferenceClient"]
    E --> F{"missing_text?"}
    F -->|Yes| G["Multi-turn QA only\ndense_dump disabled"]
    F -->|No| H{"dense_dump_prob\ntriggers?"}
    H -->|Yes| I["Single-turn dense dump\n(all valid words)"]
    H -->|No| J["Multi-turn QA\n(≤100 pairs, 4 families)"]
    G --> D
    I --> D
    J --> D
    D --> K["worker shards\n*_worker*.jsonl"]
    K --> L["merge_output_shards()\n(Driver)"]
    L --> M["Final JSONL"]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["HFDatasetImageReaderStage\n(Driver — 1 CPU)"] --> B["OCRNemotronV2Stage\n(Worker — 8 CPU + 1 GPU, batch 32)"]
    B --> C{"run_scoring_qa?"}
    C -->|No| D["JsonlSampleWriterStage\n(Worker — 2 CPU)"]
    C -->|Yes| E["OCRScoringQAStage\n(Worker — 1 CPU, batch 16)\n+ NVInferenceClient"]
    E --> F{"missing_text?"}
    F -->|Yes| G["Multi-turn QA only\ndense_dump disabled"]
    F -->|No| H{"dense_dump_prob\ntriggers?"}
    H -->|Yes| I["Single-turn dense dump\n(all valid words)"]
    H -->|No| J["Multi-turn QA\n(≤100 pairs, 4 families)"]
    G --> D
    I --> D
    J --> D
    D --> K["worker shards\n*_worker*.jsonl"]
    K --> L["merge_output_shards()\n(Driver)"]
    L --> M["Final JSONL"]

Loading

_{Reviews (10): Last reviewed commit: "Merge branch 'main' into codex/docs-nemo..." | Re-trigger Greptile}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-docs-nemotron-ocr-pipeline.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-images
• nemo/curator/curate-images/tutorials
• nemo/curator/curate-text/synthetic
• nemo/curator/curate-text/synthetic/llm-client
• nemo/curator/main/curate-text/synthetic/nemotron-ocr
• nemo/curator/main/curate-text/synthetic/nemotron-ocr/tutorial

[lbliii]

@abhinavg4 friendly ping on this docs PR for the #2118 release-docs workstream — when you have a moment, could you take a look at the Nemotron OCR pipeline guide?

Docs preview: https://nvidia-preview-codex-docs-nemotron-ocr-pipeline.docs.buildwithfern.com/nemo/curator

Thanks!

[lbliii]

@thomasdhc, since you approved the underlying Nemotron OCR implementation in #1899, could you please review this pipeline guide when you have a moment? Thank you!

[suiyoubi]

lgtm

[lbliii]

/ok to test afb0434