diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000..a168c73455 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,41 @@ +# Code of Conduct + +## Our Pledge + +The NeMo Curator project is committed to providing a welcoming, respectful, and harassment-free experience for everyone who participates in our community — in issues, pull requests, discussions, code reviews, events, and any other project space. + +## Our Standard + +This project adopts the **[Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/)** as its Code of Conduct. The full text is published at the link above and is incorporated here by reference. + +In short: + +- **Be respectful.** Treat other contributors with empathy, kindness, and patience. Assume good intent and prefer constructive feedback. +- **Be inclusive.** Welcome newcomers and people from all backgrounds. Use language that everyone can engage with. +- **Be collaborative.** Disagreements happen — keep them about the work, not the person. Defer to maintainers on direction. +- **Be accountable.** Take responsibility for mistakes, apologize when appropriate, and learn from the experience. + +## Scope + +This Code of Conduct applies in all official project spaces — the GitHub repository (issues, pull requests, discussions, code review), associated chat and event spaces, and any time an individual is representing the project in public. It also applies when project participants interact in private channels if that conduct affects the safety or well-being of community members. + +## Reporting + +If you experience or witness behavior that violates this Code of Conduct, please report it to the maintainers so we can address it. We accept reports through either of the following channels: + +- **Open a private report on GitHub** by using [Report abuse](https://github.com/contact/report-abuse) and selecting *Report content to GitHub* → *I want to report abusive content or behavior* → the relevant repository or user. GitHub forwards the report to the repository's maintainers and to GitHub Trust & Safety. +- **Contact a repository maintainer directly** via GitHub (DM or `@`-mention on a private discussion). Maintainers are listed in the repository's commit history and on the [contributors page](https://github.com/NVIDIA-NeMo/Curator/graphs/contributors). + +A dedicated reporting email will be added here once confirmed with the project's open-source program office; until then, the GitHub channels above are the supported route. + +All reports will be reviewed and investigated promptly and fairly. Maintainers are obligated to respect the privacy and security of the reporter of any incident. + +## Enforcement + +Maintainers are responsible for clarifying and enforcing this Code of Conduct and will take appropriate, fair corrective action in response to any behavior they deem inappropriate, threatening, offensive, or harmful. Enforcement follows the **[Contributor Covenant Enforcement Guidelines](https://www.contributor-covenant.org/version/2/1/code_of_conduct/#enforcement-guidelines)**, which describe a graduated set of responses ranging from a private warning to a permanent ban. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). Community Impact Guidelines are inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity). + +For answers to common questions about this Code of Conduct, see the FAQ at . Translations are available at . diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3b418e424a..34192afd78 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,22 +1,65 @@ -# Checklist - -We are glad you are contributing to NeMo Curator! Before you make a PR, be sure to read over this guide in detail. -This project will only accept contributions under the project's Apache2 license. -This checklist ensures that NeMo Curator stays easy-to-use by both users and developers. -Not all steps are necessary for some contributions, so read the linked sections for more information about each item. - -- [Checklist](#checklist) - - [General principles](#general-principles) - - [Python style](#python-style) - - [Setup and Dev](#setup-and-dev) - - [Prerequisites](#prerequisites) - - [Installation](#installation) - - [Unit tests](#unit-tests) - - [Coverage](#coverage) - - [Pull Requests (PR) Guidelines](#pull-requests-pr-guidelines) - - [Updating Package Dependencies](#updating-package-dependencies) - -## General principles +# Contributing to NeMo Curator + +We're glad you're contributing to NeMo Curator. This guide explains what kinds of contributions we accept, where to ask questions, how to find a good first issue, and how to set up your environment and open a PR. + +All contributions are accepted under the project's [Apache License 2.0](LICENSE). All participants are expected to follow our [Code of Conduct](CODE_OF_CONDUCT.md). + +## Table of Contents + +- [Ways to Contribute](#ways-to-contribute) +- [Your First Contribution](#your-first-contribution) +- [Asking Questions and Discussing Changes](#asking-questions-and-discussing-changes) +- [Code of Conduct](#code-of-conduct) +- [General Principles](#general-principles) +- [Python Style](#python-style) +- [Setup and Dev](#setup-and-dev) + - [Prerequisites](#prerequisites) + - [Installation](#installation) +- [Unit Tests](#unit-tests) +- [Coverage](#coverage) +- [Pull Requests (PR) Guidelines](#pull-requests-pr-guidelines) + - [Updating Package Dependencies](#updating-package-dependencies) + +## Ways to Contribute + +Contributions of any size are welcome. Common types: + +- **Bug fixes** — reproducible issues with a failing test where practical. +- **Documentation** — fixes, clarifications, and new tutorials in `fern/` (the canonical docs source) and the in-repo `tutorials/` directory. +- **Tests** — extending unit, integration, or GPU test coverage; converting flaky tests into deterministic ones. +- **New pipeline stages** — modality-aware filters, classifiers, dedupers, loaders, or writers that fit the existing `Stage`/`Pipeline` abstractions. +- **Recipes and examples** — end-to-end curation workflows that exercise real datasets. +- **Performance work** — benchmarks, scaling improvements, or executor enhancements (please include before/after numbers). +- **Build, CI, packaging, and dependency hygiene.** + +If you're planning a larger change (new modality, new executor, breaking API change), please open a [Discussion](https://github.com/NVIDIA-NeMo/Curator/discussions) or a draft issue first so we can align on direction before you invest in code. + +## Your First Contribution + +New to the project? Start here: + +1. Browse issues labeled [`good first issue`](https://github.com/NVIDIA-NeMo/Curator/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) — these are scoped to be approachable without deep familiarity with the codebase. +2. Comment on the issue to claim it before you start work, so we don't duplicate effort. +3. Read [General Principles](#general-principles) and follow the [PR Guidelines](#pull-requests-pr-guidelines) below. + +Good starter areas if no labeled issue catches your eye: docs typos and clarifications, missing docstrings, additional unit-test coverage, and small isolated Python-stage fixes. + +## Asking Questions and Discussing Changes + +| You want to… | Where to go | +|--------------|-------------| +| Ask a "how do I…" or "is this supported?" question | [GitHub Discussions](https://github.com/NVIDIA-NeMo/Curator/discussions) | +| Propose a non-trivial feature or design change | Open a Discussion or a draft Issue describing the design first | +| Report a reproducible bug | [GitHub Issues](https://github.com/NVIDIA-NeMo/Curator/issues) using the bug template | +| Show off a recipe or pipeline | Discussions → "Show and tell" | + +These are community channels staffed on a best-effort basis; there is no support SLA. Please avoid opening Issues for usage questions — keep Issues focused on bugs and concrete proposals so the backlog stays tractable. + +## Code of Conduct + +This project follows the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating — in issues, discussions, PRs, or any other project space — you agree to abide by its terms. To report unacceptable behavior, follow the reporting instructions in `CODE_OF_CONDUCT.md`. + +## General Principles 1. **User-oriented**: make it easy for end users, even at the cost of writing more code in the background 1. **Robust**: make it hard for users to make mistakes. 1. **Reusable**: for every piece of code, think about how it can be reused in the future and make it easy to be reused. @@ -24,7 +67,7 @@ Not all steps are necessary for some contributions, so read the linked sections 1. **Legal**: if you copy even one line of code from the Internet, make sure that the code allows the license that NeMo Curator supports. Give credit and link back to the code. 1. **Sensible**: code should make sense. If you think a piece of code might be confusing, write comments. -## Python style +## Python Style We use ``ruff`` as our style guide. To fix your format run `pre-commit install && pre-commit run --all`. **Setting up pre-commit hooks locally (optional)** (one-time setup): @@ -81,7 +124,7 @@ uv sync --extra all - If project dependencies are updated, the lock file needs to be regenerated. See [Updating Package Dependencies](#updating-package-dependencies) for the full workflow. -## Unit tests +## Unit Tests Unit tests should be simple and fast. Developers should be able to run them frequently while developing without any slowdown. ``` diff --git a/README.md b/README.md index 63821fc8b0..161c5fb1a1 100644 --- a/README.md +++ b/README.md @@ -11,179 +11,185 @@ # NVIDIA NeMo Curator -**GPU-accelerated data curation for training better AI models, faster.** Scale from laptop to multi-node clusters with modular pipelines for text, images, video, and audio. +**NeMo Curator helps ML engineers and data teams build repeatable, GPU-accelerated pipelines that load, filter, deduplicate, and transform large text, image, video, and audio datasets for AI training.** Run the same pipeline on a laptop or across a multi-node Ray cluster. > *Part of the [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/products/nemo/) software suite for managing the AI agent lifecycle.* ## Updates -- 2026-04: NeMo Curator 26.04 released with Cosmos-Xenna 0.2.0 upgrade, simplified `Resources` API, and Ray 2.54. See the [release notes](https://docs.nvidia.com/nemo/curator/latest/about/release-notes). -- 2026-02: NeMo Curator 26.02 released with Ray-based pipeline architecture for all modalities — text, image, video, and audio. +- **2026-04** — NeMo Curator 26.04: Cosmos-Xenna 0.2.0 upgrade, simplified `Resources` API, Ray runtime upgrade. See the [release notes](https://docs.nvidia.com/nemo/curator/latest/about/release-notes). +- **2026-02** — NeMo Curator 26.02: Ray-based pipeline architecture for all modalities — text, image, video, and audio. -## What You Can Do +--- -| Modality | Key Capabilities | Get Started | -|----------|-----------------|-------------| -| **Text** | Deduplication • Classification • Quality Filtering • Language Detection | [Text Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/text.html) | -| **Image** | Aesthetic Filtering • NSFW Detection • Embedding Generation • Deduplication | [Image Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/image.html) | -| **Video** | Scene Detection • Clip Extraction • Motion Filtering • Deduplication | [Video Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/video.html) | -| **Audio** | ASR Transcription • Quality Assessment • WER Filtering | [Audio Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/audio.html) | +## What You Can Build -## Quick Start +| Modality | Common Operations | Guide | +|----------|-------------------|-------| +| **Text** | Deduplication, classification, quality filtering, language detection | [Text Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/text) | +| **Image** | Aesthetic filtering, NSFW detection, embedding generation, deduplication | [Image Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/image) | +| **Video** | Scene detection, clip extraction, motion filtering, deduplication | [Video Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/video) | +| **Audio** | ASR transcription, quality assessment, WER filtering | [Audio Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/audio) | -```bash -# Install for your modality -uv pip install "nemo-curator[text_cuda12]" +### Use NeMo Curator when… -# Run the quickstart example -python tutorials/quickstart.py -``` - -**Full setup:** [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html) • [Docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator) • [Tutorials](tutorials/) +- You need **repeatable curation pipelines** — not one-off notebooks or ad-hoc scripts. +- You need **GPU and distributed execution** for data-heavy stages (dedupe, classification, embedding, inference). +- You need **modality-aware building blocks** for text, image, video, or audio. +- You want **recipes that map to NVIDIA training workflows** like Nemotron and Nemotron-CC. --- -## Architecture +## Quick Start -NeMo Curator uses a modular, Ray-based pipeline architecture. Data flows through composable processing stages — each stage handles a discrete curation task (loading, filtering, deduplication, etc.) and can be configured with independent resource requirements. +Three paths, depending on what you're trying to do. Each path is self-contained. -

- NeMo Curator architecture diagram showing modular pipeline stages -

+NeMo Curator uses [`uv`](https://docs.astral.sh/uv/) for installation. Install it once: ---- +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +``` -## Features by Modality +### Path A — CPU smoke test (no GPU required) -### Text Curation +Verify your environment and run a tiny text pipeline. -Process and curate high-quality text datasets for large language model (LLM) training with multilingual support. +```bash +uv venv && source .venv/bin/activate +uv pip install "nemo-curator[text_cpu]" +python -c "import nemo_curator; print(nemo_curator.__version__)" +``` -| Category | Features | Documentation | -|----------|----------|---------------| -| **Data Sources** | Common Crawl • Wikipedia • ArXiv • Custom datasets | [Load Data](https://docs.nvidia.com/nemo/curator/latest/curate-text/load-data/index.html) | -| **Quality Filtering** | 30+ heuristic filters • fastText classification • GPU-accelerated classifiers for domain, quality, safety, and content type | [Quality Assessment](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/quality-assessment/heuristic.html) | -| **Deduplication** | Exact • Fuzzy (MinHash LSH) • Semantic (GPU-accelerated) | [Deduplication](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/deduplication/index.html) | -| **Processing** | Text cleaning • Language identification | [Content Processing](https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/content-processing/text-cleaning.html) | +### Path B — GPU text pipeline (CUDA 12, supported Linux) ---- +The bundled quickstart starts Ray, downloads a Hugging Face model, and runs a sentiment classification pipeline on GPU. -### Image Curation +**Prerequisites:** CUDA 12 toolkit, NVIDIA driver supporting CUDA 12, Linux x86_64, ~16 GB GPU memory, network access to Hugging Face. + +```bash +uv venv && source .venv/bin/activate +uv pip install "nemo-curator[text_cuda12]" +python tutorials/quickstart.py +``` -Curate large-scale image datasets for vision language models (VLMs) and generative AI training. +### Path C — Docker (recommended for video and audio) -| Category | Features | Documentation | -|----------|----------|---------------| -| **Data Loading** | WebDataset format • Large-scale image-text pairs | [Load Data](https://docs.nvidia.com/nemo/curator/latest/curate-images/load-data/index.html) | -| **Embeddings** | CLIP embeddings for semantic analysis | [Embeddings](https://docs.nvidia.com/nemo/curator/latest/curate-images/process-data/embeddings/index.html) | -| **Filtering** | Aesthetic quality scoring • NSFW detection | [Filters](https://docs.nvidia.com/nemo/curator/latest/curate-images/process-data/filters/index.html) | +Video and audio pipelines depend on system codec libraries; the published container ships them preconfigured. + +- Container: [nemo-curator on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator) +- Setup instructions: [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/installation) + +**Full setup for all paths:** [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/installation) • [Tutorials](tutorials/) --- -### Video Curation +## Why NeMo Curator -Process large-scale video corpora with distributed, GPU-accelerated pipelines for world foundation models (WFMs). +### Proven at scale: Nemotron -| Category | Features | Documentation | -|----------|----------|---------------| -| **Data Loading** | Local paths • S3-compatible storage • HTTP(S) URLs | [Load Data](https://docs.nvidia.com/nemo/curator/latest/curate-video/load-data/index.html) | -| **Clipping** | Fixed-stride splitting • Scene-change detection (TransNetV2) | [Clipping](https://docs.nvidia.com/nemo/curator/latest/curate-video/process-data/clipping.html) | -| **Processing** | GPU H.264 encoding • Frame extraction • Motion filtering • Aesthetic filtering | [Processing](https://docs.nvidia.com/nemo/curator/latest/curate-video/process-data/filtering.html) | -| **Embeddings** | Cosmos-Embed1 for clip-level embeddings | [Embeddings](https://docs.nvidia.com/nemo/curator/latest/curate-video/process-data/embeddings.html) | -| **Deduplication** | K-means clustering • Pairwise similarity for near-duplicates | [Deduplication](https://docs.nvidia.com/nemo/curator/latest/curate-video/process-data/dedup.html) | +NeMo Curator powers the data pipelines behind [NVIDIA Nemotron](https://developer.nvidia.com/nemotron) models. The [Nemotron-4 pre-training dataset](https://arxiv.org/abs/2402.16819) was curated using NeMo Curator's text pipeline across 8+ trillion tokens of multilingual web data — quality filtering, deduplication, and domain classification at scale. ---- +The [Nemotron-CC curation pipeline](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data/curation/nemotron-cc) uses NeMo Curator end-to-end — from Common Crawl extraction through language ID, exact/fuzzy/substring deduplication, ensemble quality classification, and LLM-based synthetic data generation — to reproduce the [Nemotron-CC datasets](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2). The SDG stage is available as an [in-repo tutorial](tutorials/synthetic/nemotron_cc/). -### Audio Curation +### Benchmark results -Prepare high-quality speech datasets for automatic speech recognition (ASR) and multimodal AI training. +NeMo Curator leverages NVIDIA RAPIDS™ (cuDF, cuML, cuGraph) with Ray to scale across multi-node, multi-GPU environments. Numbers below are from the throughput study published in the [scaling docs](https://docs.nvidia.com/nemo/curator/latest/about/concepts/scaling/throughput); see the source for full methodology, software versions, and baselines. -| Category | Features | Documentation | -|----------|----------|---------------| -| **Data Loading** | Local files • Custom manifests • Public datasets (FLEURS) | [Load Data](https://docs.nvidia.com/nemo/curator/latest/curate-audio/load-data/index.html) | -| **ASR Processing** | NeMo Framework pretrained models • Automatic transcription | [ASR Inference](https://docs.nvidia.com/nemo/curator/latest/curate-audio/process-data/asr-inference/index.html) | -| **Quality Assessment** | Word Error Rate (WER) calculation • Duration analysis • Quality-based filtering | [Quality Assessment](https://docs.nvidia.com/nemo/curator/latest/curate-audio/process-data/quality-assessment/index.html) | -| **Integration** | Text curation workflow integration for multimodal pipelines | [Text Integration](https://docs.nvidia.com/nemo/curator/latest/curate-audio/process-data/text-integration/index.html) | +| Metric | Workload | Hardware | Baseline | NeMo Curator | +|--------|----------|---------|----------|--------------| +| Fuzzy dedupe speedup | RedPajama v2 subset | 3× H100 80 GB nodes | CPU-based alternative | 10.7 h → 0.65 h (**~16×**) | +| Total cost of ownership | RedPajama v2 subset | 3× H100 80 GB nodes | CPU-based alternative | $315 → $190 (**~40% lower**) | +| GPU scaling (1→4 nodes) | RedPajama v2 subset | 1, 2, 4 × H100 80 GB nodes | Single-node run | 2.05 h → 1.01 h → 0.50 h | ---- +> Token counts and exact subset sizes vary across published panels; treat per-panel labels in the source as authoritative. -## Why Data Curation? +### Quality improvements -High-quality training data is the single most important factor in building performant AI models. Raw datasets contain noise, duplicates, low-quality content, and potentially harmful material that degrade model performance and increase training costs. +In ablation studies using a 357M-parameter GPT model trained on curated Common Crawl data, NeMo Curator's pipeline stages — text cleaning, deduplication, and quality filtering — produced progressive improvements in zero-shot downstream task accuracy.

- Common data curation challenges: quality, deduplication, filtering, and scale + Model accuracy improvements across curation pipeline stages

-At scale, data curation is a **throughput maximization problem**. A typical pipeline chains stages with very different compute profiles — lightweight CPU tokenization, small GPU classifiers, large GPU inference models — and a naive sequential approach leaves most hardware idle most of the time. - -**Example:** Consider a pipeline with language identification (0.5B model, 1 GB VRAM, 2s/sample), tokenization (CPU-only, 1s/sample), and a 5B answer model (10 GB VRAM, 10s/sample) processing 1,000 questions on a single 102 GB GPU: +--- -| Approach | How it works | Total runtime | -|----------|-------------|---------------| -| **Sequential** | Process each sample through all stages, one at a time | ~13,000 seconds | -| **NeMo Curator** | Stream batches, auto-scale replicas per stage, overlap CPU/GPU work | ~1,000 seconds | +## How It Works -NeMo Curator achieves this by streaming data through the pipeline so all stages run concurrently, auto-balancing replicas to match each stage's throughput (2× language ID, 1× tokenizer, 10× answer model), and keeping GPU workers busy over 99% of the time after an initial warm-up period. See the [scaling concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts/scaling) for details. +NeMo Curator pipelines are composed of **stages**, each handling a discrete curation task (load, filter, dedupe, classify, transform, write). Stages stream **tasks** through the pipeline and are executed by a pluggable **executor**. ---- +- **Stages** declare their own resource requirements (CPU cores, GPU memory, replicas). +- **Pipelines** chain stages; the executor auto-scales replicas per stage to match throughput across the chain. +- **Streaming execution** overlaps CPU and GPU work so all stages run concurrently — typical pipelines keep GPU workers >99% busy after warm-up. +- **Executors** run the pipeline: the [XennaExecutor](https://docs.nvidia.com/nemo/curator/latest/api/reference/api-reference/executors/xenna-executor) (Cosmos-Xenna) is the production default, with experimental Ray-based backends also available — same pipeline definition, different runtime. +- **Modality plug-ins** (text, image, video, audio) provide ready-made stages on top of the same core abstractions. -## Proven at Scale: Nemotron +See the [scaling concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts/scaling) for an end-to-end walkthrough. -NeMo Curator powers the data pipelines behind [NVIDIA Nemotron](https://developer.nvidia.com/nemotron) models. For example, the [Nemotron-4 pre-training dataset](https://arxiv.org/abs/2402.16819) was curated using NeMo Curator's text processing pipeline across 8+ trillion tokens of multilingual web data, applying quality filtering, deduplication, and domain classification at scale. +

+ NeMo Curator architecture diagram showing modular pipeline stages +

--- -## Why NeMo Curator? - -### Performance at Scale +## Recipes and Tutorials -NeMo Curator leverages NVIDIA RAPIDS™ libraries such as cuDF, cuML, and cuGraph along with Ray to scale workloads across multi-node, multi-GPU environments. +| Recipe | What it does | +|--------|--------------| +| [Nemotron-CC end-to-end](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data/curation/nemotron-cc) | Reproduces the Nemotron-CC dataset from Common Crawl | +| [Nemotron-CC SDG](tutorials/synthetic/nemotron_cc/) | Synthetic data generation stage as an in-repo tutorial | +| [Text tutorials](tutorials/text/) | Loading, filtering, dedupe, classification | +| [Image tutorials](tutorials/image/) | WebDataset loading, CLIP embeddings, aesthetic/NSFW filtering | +| [Video tutorials](tutorials/video/) | Scene detection, clipping, motion filtering, dedupe | +| [Audio tutorials](tutorials/audio/) | ASR transcription, WER filtering, multimodal handoff | -**Proven Results:** -- **16× faster** fuzzy deduplication on 8 TB RedPajama v2 (1.78 trillion tokens) -- **40% lower** total cost of ownership (TCO) compared to CPU-based alternatives -- **Near-linear scaling** from one to four H100 80 GB nodes (2.05 hrs → 0.50 hrs) +--- -**Real-World Recipe:** The [Nemotron-CC curation pipeline](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data_curation/nemotron-cc) uses NeMo Curator end-to-end — from Common Crawl extraction through language identification, exact/fuzzy/substring deduplication, ensemble quality classification, and LLM-based synthetic data generation — to reproduce the [Nemotron-CC datasets](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2). The SDG stage is also available as an [in-repo tutorial](tutorials/synthetic/nemotron_cc/). +## Installation and Deployment -

- Performance benchmarks showing 16x speed improvement, 40% cost savings, and near-linear scaling -

+| Resource | Link | +|----------|------| +| Installation guide (CPU, GPU, Docker, source) | [docs.nvidia.com/nemo/curator/latest/get-started/installation](https://docs.nvidia.com/nemo/curator/latest/get-started/installation) | +| Container image | [nemo-curator on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator) | +| Infrastructure (Slurm, Kubernetes, multi-node) | [Infrastructure docs](https://docs.nvidia.com/nemo/curator/latest/reference/infra) | +| API reference | [API docs](https://docs.nvidia.com/nemo/curator/latest/api/reference/api-reference) | +| Concepts | [Concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts) | -### Quality Improvements +Supported Python and dependency versions are defined in [`pyproject.toml`](pyproject.toml); the Python versions are also shown on the PyPI badge above. The README does not duplicate them to avoid drift. -Data curation modules measurably improve model performance. In ablation studies using a 357M-parameter GPT model trained on curated Common Crawl data: +--- -

- Model accuracy improvements across curation pipeline stages -

+## Roadmap -**Results:** Progressive improvements in zero-shot downstream task performance through text cleaning, deduplication, and quality filtering stages. +Shipped changes are documented in the [release notes](https://docs.nvidia.com/nemo/curator/latest/about/release-notes). Planned work and feature direction are tracked in [GitHub Issues](https://github.com/NVIDIA-NeMo/Curator/issues) and [Discussions](https://github.com/NVIDIA-NeMo/Curator/discussions). --- -## Learn More +## Getting Help + +Pick the channel that matches your need — these are community channels staffed on a best-effort basis; there is no SLA. + +| You want to… | Channel | Typical response | +|--------------|---------|------------------| +| Ask a usage question, share a recipe, get design feedback | [GitHub Discussions](https://github.com/NVIDIA-NeMo/Curator/discussions) | A few business days | +| Report a reproducible bug or regression | [GitHub Issues](https://github.com/NVIDIA-NeMo/Curator/issues) — use the bug template | A few business days for triage | +| Request a feature or new modality capability | [GitHub Issues](https://github.com/NVIDIA-NeMo/Curator/issues) — use the feature-request template | Triaged into a milestone when accepted | +| Read the docs | [docs.nvidia.com/nemo/curator/latest](https://docs.nvidia.com/nemo/curator/latest) | — | -| Resource | Links | -|----------|-------| -| **Documentation** | [Main Docs](https://docs.nvidia.com/nemo/curator/latest/) • [API Reference](https://docs.nvidia.com/nemo/curator/latest/apidocs/index.html) • [Concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts/index.html) | -| **Tutorials** | [Text](tutorials/text/) • [Image](tutorials/image/) • [Video](tutorials/video/) • [Audio](tutorials/audio/) | -| **Recipes** | [Nemotron-CC: end-to-end web data curation](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data_curation/nemotron-cc) • [SDG tutorial (in-repo)](tutorials/synthetic/nemotron_cc/) | -| **Deployment** | [Installation](https://docs.nvidia.com/nemo/curator/latest/admin/installation.html) • [Infrastructure](https://docs.nvidia.com/nemo/curator/latest/reference/infrastructure/index.html) | -| **Community** | [GitHub Discussions](https://github.com/NVIDIA-NeMo/Curator/discussions) • [Issues](https://github.com/NVIDIA-NeMo/Curator/issues) | +Please do not use Issues for "how do I…" questions — they belong in Discussions so they remain searchable for other users. --- -## Contribute +## Contributing -We welcome community contributions! Please refer to [CONTRIBUTING.md](https://github.com/NVIDIA/NeMo/blob/stable/CONTRIBUTING.md) for guidelines. +Contributions are welcome — bug fixes, docs, tutorials, new stages, and tests. **See [CONTRIBUTING.md](CONTRIBUTING.md)** for the full guide, including how to pick a good first issue, set up your environment, and open a signed-off PR. All participants are expected to follow our [Code of Conduct](CODE_OF_CONDUCT.md). ---- +## License + +NeMo Curator is released under the **Apache License 2.0**. See [LICENSE](LICENSE) for the full text. ## Citation -If you find NeMo Curator useful in your research, please cite: +If you use NeMo Curator in your research, please cite: ```bibtex @misc{nemo_curator,