How nvidia/nemo-curator works
Project Overview & Main Components
• NeMo Curator is a Python library that offers modular pipelines for data curation (text and image) with support for downloading, cleaning, filtering, deduplication, and PII redaction. It is designed to operate on both CPU and GPU (leveraging Dask and RAPIDS), and it integrates easily into larger data processing workflows.
• The project’s file tree indicates clearly separated directories for core library code (in the “nemo_curator” package), support modules (like “filters”, “modifiers”, “modules”, “pii”, “synthetic”, “services”, etc.), configuration files (“config” directory), examples, tests, and tutorials. There are also extensive CLI scripts in “nemo_curator/scripts.”
Analyzing the Directory Structure for Architectural Insights
• The central package (nemo_curator) is organized into submodules such as:
• classifiers – for text/image classification (quality, domain, NSFW, aesthetic, etc.)
• datasets – for dataset handling and pipeline creation.
• download – to retrieve data from multiple sources (Common Crawl, ArXiv, Wikipedia).
• filters – for filtering mechanisms including heuristic filters and classifier-based filtering.
• modules – containing core processing functions like exact, fuzzy, and semantic deduplication, along with splitting and joining operations.
• modifiers – for data modifications (e.g., line removal, markdown remover, quotation remover) applied as part of the cleaning pipelines.
• services – for external integrations and managing model clients.
• synthetic – for synthetic data generation modules.
• tasks – to run downstream tasks and metrics.
• utils – utility functions to support configuration, file IO, GPU handling, and distributed processing.
• Additional directories like “docs,” “examples,” “tutorials,” and “tests” confirm that the project is well-documented, has a mature testing suite, and provides practical usage examples.
• Configuration files (YAML files) in “config” indicate customizable behavior across different use cases (downloaders, deduplication parameters, quality filters, etc.).
• GitHub workflows, Dockerfile, and release scripts indicate support for CI/CD and containerized deployment.
Relationships & Interactions
• The system is intrinsically modular:
• Data ingestion/download modules feed raw data into the pipelines.
• Preprocessing modules (modifiers and text cleaning) prepare the data.
• Filtering modules (heuristic, classifier-based, and language identification) then evaluate and screen the data.
• Deduplication modules (exact, fuzzy, semantic) remove redundant elements.
• PII modules handle sensitive data de-identification.
• Finally, modules can prepare datasets for downstream training or benchmarking.
Important Architectural Patterns & Design Principles
• Modularity & Extensibility: Each functional component (download, filtering, deduplication, modification, etc.) is implemented as a self-contained module in its own directory. This supports customization and easy pipeline reordering.
• Pipeline Orchestration: The “Sequential” pipeline pattern is used to chain different processing stages, as demonstrated in the README’s Python API example.
• Distributed & Parallel Processing: Integration with Dask & RAPIDS to scale the data pipelines across multiple GPUs/nodes.
• Separation of concerns: Distinct directories separate core functionality (nemo_curator) from scripts, configuration, documentation, and tests.