How modelscope/data-juicer works
Project Type and Purpose
• Data-Juicer is a Python-based, backend-oriented library and CLI/API service for systematic, large-scale data processing (text, image, audio, video) tailored to foundation models (LLMs). It provides modular “OPs” (operators), config-driven pipelines, distributed execution (Ray), analysis & export tools, and sandbox laboratory features.
High-Level Architecture Components
a. User Entry Points
• CLI (app.py / service.py) – load YAML/JSON config, invoke pipeline
• Python API (data_juicer package) – programmatic pipeline assembly
• Demo / FastAPI services (demos/api_service/app.py) – HTTP endpoints for ad hoc processing
b. Configuration Layer
• configs/… YAML files for annotation, datasets, recipes, demos
• data_juicer.config.config – central config loader & validator
c. Core Pipeline Orchestrator
• Dataset Builder (dataset_builder, dj_dataset, ray_dataset) – abstract over local/remote/parquet/JSONL sources
• Executor Factory – chooses between DefaultExecutor (single-process) or RayExecutor (distributed)
• Pipeline Definition – sequence of OPs (filter, mapper, aggregator, deduplicator, grouper, etc.) loaded by adapter & factory
d. Operator Library
• data_juicer.ops – 100+ OP modules grouped in subpackages (filter, mapper, deduplicator, aggregator, selector, fusion)
• Extensibility points via mixins, base_op
e. Execution & Monitoring
• data_juicer.core.executor – task scheduling, parallelism
• data_juicer.core.monitor – resource & progress tracking
• data_juicer.core.tracer – logging & tracing of pipeline steps
f. Analysis & Visualization
• data_juicer.analysis – diversity, overall, column-wise, custom analyses
• demos/data_visualization_… – web front-ends and Jupyter notebooks
g. Export & Format
• data_juicer.format – JSON, CSV, TSV, Parquet formatters
• Exporter module to write outputs to filesystem, cloud storage
h. Sandbox Laboratory
• data_juicer.core.sandbox – hookable environments for feedback-driven tuning, evaluators, model executors
i. Tools & Utilities
• tools/ – standalone scripts (format conversion, evaluation, HPO, distributed dedup, preprocess/postprocess, quality classifier)
• data_juicer.utils – caching, file handling, registry, model loading, resource utilities
External & Infrastructure Dependencies
• Ray cluster (for distributed execution)
• Cloud/object storage (S3/HDFS) for large datasets
• Docker (Dockerfile, GitHub workflows)
• GitHub Actions (CI/CD for tests, docs, publish to PyPI/Docker)
• PyPI repository, Docker Hub registry
Relationships & Data Flow
1. User invokes CLI/API → config loader validates YAML → dataset builder instantiates source
2. Pipeline assembler loads OP sequence per config
3. Executor schedules tasks (single‐process or Ray cluster)
4. Each data batch flows through OPs → monitor/tracer capture metrics
5. Processed data passed to analysis or exported via formatters
6. Sandbox module optionally wraps model inference + feedback loops
7. Final outputs persisted; logs & metrics sent to monitoring systems
Architectural Patterns & Principles
• Modular plugin architecture for OPs & adapters
• Factory pattern for executor & dataset strategies
• Config-driven pipelines for reproducibility
• Separation of concerns: ingestion, transformation (ops), execution, export, analysis
• Extensibility via sandbox hooks & developer guide