How modelscope/easydistill works
EasyDistill 2 is a config-driven toolkit for converting black-box teacher models into training data. Users select a workflow through the CLI; the configured pipeline applies generation and, as appropriate, synthesis, evaluation/filtering, rewriting, balancing, or preference scoring, then builds SFT or DPO outputs. Separate capabilities cover text, vision-language, image/video generation, and agent trajectories. The graph preserves these documented branches, while sampled source confirms selected dispatch, generation, backend, and data-model relationships; unsampled implementations are represented at their existing directory paths, with links omitted where evidence is insufficient.
Main entry points: pipeline Distillation pipelines · runners Job runners · backend_factory.py Backend factory · generation.py Generation operators
Components 22
Workflow entry
- CLI dispatch
easydistill/cli/main.py - Job runners
easydistill/cli/runners - Dataset loading
easydistill/cli/data_loaders.py
Distillation processing
- Distillation pipelines
easydistill/pipeline - Basic distillation
easydistill/basic/distillation.py - Generation operators
easydistill/operators/generation.py - Synthesis and rewrite
easydistill/rewrite - Evaluation and filtering
easydistill/eval - Agent trajectories
easydistill/operators/agent/trajectory.py - Search task evolution
easydistill/operators/search_agent/task_evolver.py - Preference construction
easydistill/operators/preference
Multimodal generation
- Image generation
easydistill/operators/t2i - Video generation
easydistill/operators/t2v/t2v_generation.py
Models and services
- Backend factory
easydistill/cli/backend_factory.py - Model backend API
easydistill/backends/base.py - Image backends
easydistill/backends - Video backends
easydistill/backends/t2v_base.py
Training data
- Typed data models
easydistill/data/models.py - SFT dataset builder
easydistill/operators/sft_builder.py
Other components
- Workflow user
- Teacher model service
- Training-ready datasets
Connections 27
- Workflow user → CLI dispatch: selects job
- CLI dispatch → Job runners: dispatches job
easydistill/cli/main.py - Job runners → Backend factory: builds backend
easydistill/cli/runners/pipeline.py - Job runners → Distillation pipelines: runs workflow
easydistill/cli/runners/pipeline.py - Job runners → Basic distillation: runs distillation
- Job runners → Dataset loading: loads input data
- Distillation pipelines → Generation operators: composes generation
- Distillation pipelines → Synthesis and rewrite: composes synthesis
easydistill/pipeline/common.py - Distillation pipelines → Evaluation and filtering: composes evaluation
- Distillation pipelines → SFT dataset builder: builds SFT
- Distillation pipelines → Preference construction: builds DPO pairs
easydistill/pipeline/common.py - Distillation pipelines → Agent trajectories: rolls trajectories
- Distillation pipelines → Search task evolution: evolves search tasks
- Distillation pipelines → Image generation: supports image jobs
easydistill/pipeline/common.py - Distillation pipelines → Video generation: supports video jobs
- Backend factory → Model backend API: creates backend
easydistill/cli/backend_factory.py - Generation operators → Model backend API: requests generation
easydistill/operators/generation.py - Model backend API → Teacher model service: calls model service
- Backend factory → Image backends: creates image backend
easydistill/cli/backend_factory.py - Image generation → Image backends: requests images
- Backend factory → Video backends: creates video backend
- Video generation → Video backends: requests videos
easydistill/operators/t2v/t2v_generation.py - Generation operators → Typed data models: uses request models
easydistill/operators/generation.py - SFT dataset builder → Typed data models: builds samples
- Preference construction → Typed data models: builds preference samples
- SFT dataset builder → Training-ready datasets: exports datasets
- Preference construction → Training-ready datasets: exports preference pairs