# huggingface/datatrove architecture

> Architecture diagram of the GitHub repository huggingface/datatrove, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/datatrove
- Repository: https://github.com/huggingface/datatrove

## Overview

##### Main Components and Their Purpose
- Data Ingestion/Input:
– Represent the “Readers” (found in src/datatrove/pipeline/readers and examples) that ingest data from various sources (local files, S3, HF datasets, etc via fsspec). Label components like “CSVReader,” “HuggingFace Reader,” “WARC Reader” (if applicable).
- Processing Pipeline Blocks:
– Group together the various pipeline stages (located in src/datatrove/pipeline) by type:
○ Readers – those that ingest and convert raw data into the standardized Document format.
○ Extractors – e.g., the trafilatura-based extractor (src/datatrove/pipeline/extractors/trafilatura.py).
○ Filters – including language filters, regex filters, quality filters (src/datatrove/pipeline/filters).
○ Deduplication Blocks – such as bloom_filter, exact_substrings, minhash, and sentence-level deduplication in src/datatrove/pipeline/dedup.
○ Tokenizers – blocks for tokenization and related statistics present in src/datatrove/pipeline/tokens.
○ Writers – modules that save processed Documents to disk or cloud storage (src/datatrove/pipeline/writers).
- Statistics/Logging Modules:
– Include components that calculate statistics or collect logging data (located in src/datatrove/pipeline/stats and the logging configuration files). Also highlight the “completions” tracking in the logging directories.
- Execution Environment (Orchestration):
– Present the executors (LocalPipelineExecutor and SlurmPipelineExecutor in src/datatrove/executor) that run the pipelines on different resource setups. This should include task distribution, the concept of ‘workers’ (for multiprocessing) and support for multi-node operations.
- Supporting Modules and Assets:
– Show assets such as banned words lists and tokenizer assignments (located under src/datatrove/assets).
– Utilities for data, hashing, and tokenization (src/datatrove/utils) which support pipeline operations.

##### Relationships and Data Flow
- After the processing blocks, illustrate the writing/outcome branch where cleaned/deduplicated Documents are persisted—this is the “Writers” component.
- In parallel, highlight the logging/statistics flow: as Documents go through each stage, statistics are collected and written into a designated “stats” and “logs” directory.
- How the “Executor” orchestrates the whole process – it orchestrates the parallel execution (which includes sharding the dataset across tasks and assigning a number of workers per task) and interacts with the file system for tracking completion.

##### Architectural Patterns
- Modular Pipeline: Emphasize that each pipeline block is independent (generator-based processing) allowing flexible reconfiguration and extension.
- Distributed/Parallel Execution: Highlight the design principles behind task sharding and the execution across local or slurm-controlled environments.
- Extensibility: Note that new blocks can be inserted (using custom blocks or functions) and that the entire system is built to process “Documents” in a standard format.

##### Technologies and Frameworks
- Python as the main language with modern packaging (pyproject.toml).
- Continuous integration via GitHub workflows (as seen in .github/workflows).
- External libraries: fsspec (for file-system abstraction), trafilatura (for extraction), and others used in specific pipeline blocks.
- Multiprocessing and Slurm job arrays for parallel execution.
- Pre-commit hooks and configuration files to support development.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% External Data Sources and Ingestion
  A["Data Storage (Raw Input)"]:::storage
  B["Reader Blocks"]:::ingestion
  A -->|"reads_raw_data"| B

  %% Processing Pipeline
  subgraph "Processing Pipeline"
    direction TB
    D["Trafilatura Extractor"]:::pipeline
    E["Filter Blocks"]:::pipeline
    subgraph "Deduplication Blocks"
      direction TB
      F1["Bloom Filter"]:::pipeline
      F2["Exact Substrings"]:::pipeline
      F3["Minhash"]:::pipeline
      F4["Sentence Dedup"]:::pipeline
      F5["URL Dedup"]:::pipeline
    end
    G["Tokenizer Blocks"]:::pipeline
    H["Writer Blocks"]:::pipeline
  end

  B -->|"converts_to_Document"| D
  D -->|"extracts_content"| E
  E -->|"filters_out_noise"| F1
  F1 -->|"passes_data"| F2
  F2 -->|"removes_duplicates"| F3
  F3 -->|"refines_duplicates"| F4
  F4 -->|"final_deduplication"| F5
  F5 -->|"prepares_for_tokenization"| G
  G -->|"formats_tokens"| H
  H -->|"saves_processed_Documents"| O["Persistent Storage"]:::storage

  %% Logging & Statistics
  I["Logging & Statistics"]:::logging
  D ---|"logs_stats"| I
  E ---|"logs_stats"| I
  F1 ---|"logs_stats"| I
  F2 ---|"logs_stats"| I
  F3 ---|"logs_stats"| I
  F4 ---|"logs_stats"| I
  F5 ---|"logs_stats"| I
  G ---|"logs_stats"| I
  H ---|"logs_stats"| I

  %% Execution Environment
  subgraph "Executor"
    direction TB
    J["LocalPipelineExecutor"]:::executor
    K["SlurmPipelineExecutor"]:::executor
  end
  J ---|"orchestrates"| B
  K ---|"orchestrates"| B

  %% Supporting Modules
  L["Assets"]:::utility
  M["Utilities"]:::utility
  N["Usage Examples"]:::utility
  N ---|"demonstrates_usage"| B

  %% Legend (for color and shape coding)
  subgraph "Legend"
    direction TB
    Legend1["Ingestion: Light Teal"] 
    Legend2["Pipeline: Light Orange"]
    Legend3["Orchestration: Light Green"]
    Legend4["Logging: Light Red"]
    Legend5["Storage: Light Purple"]
    Legend6["Utility: Light Pink"]
  end

  %% Click Events for Component Mapping
  click B "https://github.com/huggingface/datatrove/tree/main/src/datatrove/pipeline/readers"
  click D "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/extractors/trafilatura.py"
  click E "https://github.com/huggingface/datatrove/tree/main/src/datatrove/pipeline/filters"
  click F1 "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/bloom_filter.py"
  click F2 "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/exact_substrings.py"
  click F3 "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/minhash.py"
  click F4 "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/sentence_dedup.py"
  click F5 "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/dedup/url_dedup.py"
  click G "https://github.com/huggingface/datatrove/blob/main/src/datatrove/pipeline/tokens/tokenizer.py"
  click H "https://github.com/huggingface/datatrove/tree/main/src/datatrove/pipeline/writers"
  click I "https://github.com/huggingface/datatrove/tree/main/src/datatrove/pipeline/stats"
  click J "https://github.com/huggingface/datatrove/blob/main/src/datatrove/executor/local.py"
  click K "https://github.com/huggingface/datatrove/blob/main/src/datatrove/executor/slurm.py"
  click L "https://github.com/huggingface/datatrove/tree/main/src/datatrove/assets"
  click M "https://github.com/huggingface/datatrove/tree/main/src/datatrove/utils"
  click N "https://github.com/huggingface/datatrove/blob/main/examples/filter_hf_dataset.py"

  %% Styles
  classDef ingestion fill:#D1E8E2,stroke:#333,stroke-width:2px;
  classDef pipeline fill:#FAD7A0,stroke:#333,stroke-width:2px;
  classDef executor fill:#A9DFBF,stroke:#333,stroke-width:2px;
  classDef logging fill:#F5B7B1,stroke:#333,stroke-width:2px;
  classDef storage fill:#D7BDE2,stroke:#333,stroke-width:2px;
  classDef utility fill:#FADBD8,stroke:#333,stroke-width:2px;
```
