# modelscope/data-juicer architecture

> Architecture diagram of the GitHub repository modelscope/data-juicer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modelscope/data-juicer
- Repository: https://github.com/modelscope/data-juicer

## Overview

##### Project Type and Purpose
- Data-Juicer is a Python-based, backend-oriented library and CLI/API service for systematic, large-scale data processing (text, image, audio, video) tailored to foundation models (LLMs). It provides modular “OPs” (operators), config-driven pipelines, distributed execution (Ray), analysis & export tools, and sandbox laboratory features.

##### High-Level Architecture Components
a. User Entry Points
- CLI (app.py / service.py) – load YAML/JSON config, invoke pipeline
- Python API (data_juicer package) – programmatic pipeline assembly
- Demo / FastAPI services (demos/api_service/app.py) – HTTP endpoints for ad hoc processing
b. Configuration Layer
- configs/… YAML files for annotation, datasets, recipes, demos
- data_juicer.config.config – central config loader & validator
c. Core Pipeline Orchestrator
- Dataset Builder (dataset_builder, dj_dataset, ray_dataset) – abstract over local/remote/parquet/JSONL sources
- Executor Factory – chooses between DefaultExecutor (single-process) or RayExecutor (distributed)
- Pipeline Definition – sequence of OPs (filter, mapper, aggregator, deduplicator, grouper, etc.) loaded by adapter & factory
d. Operator Library
- data_juicer.ops – 100+ OP modules grouped in subpackages (filter, mapper, deduplicator, aggregator, selector, fusion)
- Extensibility points via mixins, base_op
e. Execution & Monitoring
- data_juicer.core.executor – task scheduling, parallelism
- data_juicer.core.monitor – resource & progress tracking
- data_juicer.core.tracer – logging & tracing of pipeline steps
f. Analysis & Visualization
- data_juicer.analysis – diversity, overall, column-wise, custom analyses
- demos/data_visualization_… – web front-ends and Jupyter notebooks
g. Export & Format
- data_juicer.format – JSON, CSV, TSV, Parquet formatters
- Exporter module to write outputs to filesystem, cloud storage
h. Sandbox Laboratory
- data_juicer.core.sandbox – hookable environments for feedback-driven tuning, evaluators, model executors
i. Tools & Utilities
- tools/ – standalone scripts (format conversion, evaluation, HPO, distributed dedup, preprocess/postprocess, quality classifier)
- data_juicer.utils – caching, file handling, registry, model loading, resource utilities

##### External & Infrastructure Dependencies
- Ray cluster (for distributed execution)
- Cloud/object storage (S3/HDFS) for large datasets
- Docker (Dockerfile, GitHub workflows)
- GitHub Actions (CI/CD for tests, docs, publish to PyPI/Docker)
- PyPI repository, Docker Hub registry

##### Relationships & Data Flow
1. User invokes CLI/API → config loader validates YAML → dataset builder instantiates source
2. Pipeline assembler loads OP sequence per config
3. Executor schedules tasks (single‐process or Ray cluster)
4. Each data batch flows through OPs → monitor/tracer capture metrics
5. Processed data passed to analysis or exported via formatters
6. Sandbox module optionally wraps model inference + feedback loops
7. Final outputs persisted; logs & metrics sent to monitoring systems

##### Architectural Patterns & Principles
- Modular plugin architecture for OPs & adapters
- Factory pattern for executor & dataset strategies
- Config-driven pipelines for reproducibility
- Separation of concerns: ingestion, transformation (ops), execution, export, analysis
- Extensibility via sandbox hooks & developer guide

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Layer
    subgraph "Entry Layer"
        CLI["CLI/API Entry"]:::entry
        API["Python API"]:::entry
        Demo["Demo / FastAPI Service"]:::entry
    end

    %% Core Pipeline
    subgraph "Core Pipeline" 
        Config["Config Loader"]:::core
        subgraph "Dataset Builder"
            DB1["dataset_builder"]:::core
            DB2["dj_dataset"]:::core
            DB3["ray_dataset"]:::core
            DB4["load_strategy"]:::core
        end
        Adapter["Pipeline Adapter"]:::core
        subgraph "Executor Factory & Executors"
            EF["Executor Factory"]:::core
            DE["DefaultExecutor"]:::core
            RE["RayExecutor"]:::core
            BE["BaseExecutor"]:::core
        end
        Monitor["Monitor"]:::core
        Tracer["Tracer"]:::core
    end

    %% Extensions & Tools
    subgraph "Extensions & Tools"
        OP["OP Library"]:::ext
        Analysis["Analysis Module"]:::ext
        Format["Formatter & Exporter"]:::ext
        Sandbox["Sandbox Laboratory"]:::ext
        subgraph "Tools & Utilities"
            Utils["Utils"]:::ext
            Tools["Standalone Tools"]:::ext
        end
    end

    %% External Systems
    subgraph "External"
        RayCluster[(Ray Cluster)]:::external
        Storage[(Storage)]:::external
        DockerCI[(Docker Hub & CI/CD)]:::external
    end

    %% Data Flow & Dependencies
    User[("User")] -->|invoke| CLI
    User -->|invoke| API
    User -->|HTTP request| Demo

    CLI -->|control| Config
    API -->|control| Config
    Demo -->|control| Config

    Config -->|instantiates| DB1
    DB1 --> DB2
    DB2 --> DB3
    DB3 --> DB4

    DB4 -->|loads pipeline| Adapter
    Adapter -->|creates| EF
    EF -->|selects| DE
    EF -->|selects| RE
    EF -->|extends| BE

    DE -->|executes OPs| OP
    RE -->|executes OPs on| RayCluster
    OP -->|uses| DE
    OP -->|uses| RE

    DE -->|reports| Monitor
    RE -->|reports| Monitor
    DE -->|traces| Tracer
    RE -->|traces| Tracer

    OP -->|produces data| Analysis
    OP -->|produces data| Format
    Sandbox -->|wraps| OP
    Analysis -->|visualizes| Storage
    Format -->|writes to| Storage
    Tools -->|utility scripts| Storage

    %% External interactions
    DockerCI -->|deploys| Demo
    DockerCI -->|builds| CLI

    %% Click Events
    click CLI "https://github.com/modelscope/data-juicer/blob/main/app.py"
    click CLI "https://github.com/modelscope/data-juicer/blob/main/service.py"
    click API "https://github.com/modelscope/data-juicer/blob/main/data_juicer/__init__.py"
    click API "https://github.com/modelscope/data-juicer/blob/main/data_juicer/config/config.py"
    click Demo "https://github.com/modelscope/data-juicer/blob/main/demos/api_service/app.py"
    click Demo "https://github.com/modelscope/data-juicer/blob/main/demos/api_service/utils.py"
    click Demo "https://github.com/modelscope/data-juicer/blob/main/demos/api_service/wrapped_filters.py"
    click Demo "https://github.com/modelscope/data-juicer/blob/main/demos/api_service/wrapped_mappers.py"
    click Config "https://github.com/modelscope/data-juicer/blob/main/data_juicer/config/config.py"
    click DB1 "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/data/dataset_builder.py"
    click DB2 "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/data/dj_dataset.py"
    click DB3 "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/data/ray_dataset.py"
    click DB4 "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/data/load_strategy.py"
    click EF "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/executor/factory.py"
    click DE "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/executor/default_executor.py"
    click RE "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/executor/ray_executor.py"
    click BE "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/executor/base.py"
    click Adapter "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/adapter.py"
    click Monitor "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/monitor.py"
    click Tracer "https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/tracer.py"
    click OP "https://github.com/modelscope/data-juicer/tree/main/data_juicer/ops/"
    click Analysis "https://github.com/modelscope/data-juicer/tree/main/data_juicer/analysis/"
    click Format "https://github.com/modelscope/data-juicer/tree/main/data_juicer/format/"
    click Sandbox "https://github.com/modelscope/data-juicer/tree/main/data_juicer/core/sandbox/"
    click Utils "https://github.com/modelscope/data-juicer/tree/main/data_juicer/utils/"
    click Tools "https://github.com/modelscope/data-juicer/tree/main/tools/"
    click DockerCI "https://github.com/modelscope/data-juicer/tree/main/Dockerfile"
    click DockerCI "https://github.com/modelscope/data-juicer/tree/main/.github/workflows/"
    click DockerCI "https://github.com/modelscope/data-juicer/blob/main/pyproject.toml"

    %% Styles
    classDef entry fill:#ADD8E6,stroke:#000,color:#000;
    classDef core fill:#90EE90,stroke:#000,color:#000;
    classDef ext fill:#FFD700,stroke:#000,color:#000;
    classDef external fill:#D3D3D3,stroke:#000,color:#000;
```
