# nvidia/nemo-curator architecture

> Architecture diagram of the GitHub repository nvidia/nemo-curator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/nemo-curator
- Repository: https://github.com/nvidia/nemo-curator

## Overview

##### Project Overview & Main Components
- NeMo Curator is a Python library that offers modular pipelines for data curation (text and image) with support for downloading, cleaning, filtering, deduplication, and PII redaction. It is designed to operate on both CPU and GPU (leveraging Dask and RAPIDS), and it integrates easily into larger data processing workflows.
- The project’s file tree indicates clearly separated directories for core library code (in the “nemo_curator” package), support modules (like “filters”, “modifiers”, “modules”, “pii”, “synthetic”, “services”, etc.), configuration files (“config” directory), examples, tests, and tutorials. There are also extensive CLI scripts in “nemo_curator/scripts.”

##### Analyzing the Directory Structure for Architectural Insights
- The central package (nemo_curator) is organized into submodules such as:
- classifiers – for text/image classification (quality, domain, NSFW, aesthetic, etc.)
- datasets – for dataset handling and pipeline creation.
- download – to retrieve data from multiple sources (Common Crawl, ArXiv, Wikipedia).
- filters – for filtering mechanisms including heuristic filters and classifier-based filtering.
- modules – containing core processing functions like exact, fuzzy, and semantic deduplication, along with splitting and joining operations.
- modifiers – for data modifications (e.g., line removal, markdown remover, quotation remover) applied as part of the cleaning pipelines.
- services – for external integrations and managing model clients.
- synthetic – for synthetic data generation modules.
- tasks – to run downstream tasks and metrics.
- utils – utility functions to support configuration, file IO, GPU handling, and distributed processing.
- Additional directories like “docs,” “examples,” “tutorials,” and “tests” confirm that the project is well-documented, has a mature testing suite, and provides practical usage examples.
- Configuration files (YAML files) in “config” indicate customizable behavior across different use cases (downloaders, deduplication parameters, quality filters, etc.).
- GitHub workflows, Dockerfile, and release scripts indicate support for CI/CD and containerized deployment.

##### Relationships & Interactions
- The system is intrinsically modular:
- Data ingestion/download modules feed raw data into the pipelines.
- Preprocessing modules (modifiers and text cleaning) prepare the data.
- Filtering modules (heuristic, classifier-based, and language identification) then evaluate and screen the data.
- Deduplication modules (exact, fuzzy, semantic) remove redundant elements.
- PII modules handle sensitive data de-identification.
- Finally, modules can prepare datasets for downstream training or benchmarking.

##### Important Architectural Patterns & Design Principles
- Modularity & Extensibility: Each functional component (download, filtering, deduplication, modification, etc.) is implemented as a self-contained module in its own directory. This supports customization and easy pipeline reordering.
- Pipeline Orchestration: The “Sequential” pipeline pattern is used to chain different processing stages, as demonstrated in the README’s Python API example.
- Distributed & Parallel Processing: Integration with Dask & RAPIDS to scale the data pipelines across multiple GPUs/nodes.
- Separation of concerns: Distinct directories separate core functionality (nemo_curator) from scripts, configuration, documentation, and tests.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion & Extraction Subsystem
    subgraph "Data Ingestion & Extraction"
        DI["Download Modules"]:::ingestion
    end

    %% Data Preprocessing & Cleaning Subsystem
    subgraph "Data Preprocessing & Cleaning"
        PP["Modifiers"]:::processing
    end

    %% Filtering & Classification Subsystem
    subgraph "Filtering & Classification"
        TC["Text Classification"]:::filtering
        IC["Image Classification"]:::filtering
        FM["Filtering Modules"]:::filtering
    end

    %% Deduplication & Consolidation Subsystem
    subgraph "Deduplication & Consolidation"
        ED["Exact Deduplication"]:::dedup
        FDm["Fuzzy Dedup (Module)"]:::dedup
        FDs["Fuzzy Dedup (Script)"]:::dedup
        SDm["Semantic Dedup (Module)"]:::dedup
        SDs["Semantic Dedup (Script)"]:::dedup
    end

    %% PII Processing Subsystem
    subgraph "PII Processing"
        PII["PII Identification & Removal"]:::pii
    end

    %% Dataset Handling Subsystem
    subgraph "Dataset Handling"
        DS["Dataset Pipeline"]:::dataset
    end

    %% Pipeline Orchestration Subsystem
    subgraph "Pipeline Orchestration"
        PE["Pipeline Execution (Library)"]:::orchestration
        PEx["Pipeline Execution (Example)"]:::orchestration
        CLI["CLI Scripts"]:::orchestration
    end

    %% External Services & Utilities
    subgraph "External Services & Utilities"
        ES["External Services"]:::external
        SYN["Synthetic Data Generation"]:::external
        UT["Utility Functions"]:::external
        CM["Configuration Management"]:::config
        Dask["Dask"]:::external
        RAPIDS["RAPIDS"]:::external
        fastText["fastText"]:::external
    end

    %% Main Data Flow Connections
    DI -->|"raw_data"| PP
    PP -->|"clean_data"| TC
    PP -->|"clean_data"| IC
    PP -->|"clean_data"| FM
    TC -->|"classification"| FM
    IC -->|"classification"| FM
    FM -->|"filtered_data"| ED
    ED --> FDm
    FDm --> FDs
    FDs --> SDm
    SDm --> SDs
    SDs -->|"deduped_data"| PII
    PII -->|"anonymized_data"| DS
    DS -->|"curated_dataset"| PE
    PE --> PEx
    PEx --> CLI

    %% External Dependency Connections (Dotted Lines)
    ES --- TC
    ES --- IC
    ES --- FM

    Dask --- FDm
    RAPIDS --- SDm
    fastText --- TC

    UT --- PP
    UT --- FM

    CM --- DI
    CM --- PP
    CM --- FM

    %% Click Events
    click DI "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/download"
    click CM "https://github.com/nvidia/nemo-curator/tree/main/config"
    click PP "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/modifiers"
    click TC "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/classifiers"
    click IC "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/image/classifiers"
    click FM "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/filters"
    click ED "https://github.com/nvidia/nemo-curator/blob/main/nemo_curator/modules/exact_dedup.py"
    click FDm "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/modules/fuzzy_dedup"
    click FDs "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/scripts/fuzzy_deduplication"
    click SDm "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/modules/semantic_dedup"
    click SDs "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/scripts/semdedup"
    click PII "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/pii"
    click DS "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/datasets"
    click PE "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/nemo_run"
    click PEx "https://github.com/nvidia/nemo-curator/tree/main/examples/nemo_run"
    click CLI "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/scripts"
    click ES "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/services"
    click SYN "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/synthetic"
    click UT "https://github.com/nvidia/nemo-curator/tree/main/nemo_curator/utils"

    %% Class Definitions for Colors and Styles
    classDef ingestion fill:#d0f0c0,stroke:#333,stroke-width:2px;
    classDef processing fill:#a7d5f8,stroke:#333,stroke-width:2px;
    classDef filtering fill:#ffefba,stroke:#333,stroke-width:2px;
    classDef dedup fill:#f9c0c0,stroke:#333,stroke-width:2px;
    classDef pii fill:#fcd5ce,stroke:#333,stroke-width:2px;
    classDef dataset fill:#d6cbd3,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#c5e1a5,stroke:#333,stroke-width:2px;
    classDef external fill:#ffe0b2,stroke:#333,stroke-width:2px;
    classDef config fill:#cdeac0,stroke:#333,stroke-width:2px;
```
