alibaba/roll

How alibaba/roll works

ROLL is a distributed library for large-scale LLM reinforcement-learning and post-training. Its principal workflow configures a training run, prepares data, then runs a selected pipeline across distributed workers to produce trained or aligned models. Major documented branches include SFT, preference optimization, distillation, RL with verifiable rewards, agentic interaction, and diffusion training. The graph includes shared model, scheduling, and checkpoint layers. Some pipeline-to-runtime links are documented or indexed, but unsampled implementation details are left unverified rather than inferred.

Main entry points: base_pipeline.py Pipeline Base · collator.py Dataset Collation · model_providers.py Model Providers · agentic_pipeline.py Agentic RL

Components 19

Training Pipelines

SFT roll/pipeline/sft/sft_pipeline.py
Preference Tuning roll/pipeline/dpo/dpo_pipeline.py
Distillation roll/pipeline/distill/distill_pipeline.py
Verifiable RL roll/pipeline/rlvr/rlvr_pipeline.py
Diffusion Training roll/pipeline/diffusion/diffusion_pipeline.py
DMD Distillation roll/pipeline/diffusion/dmd/dmd_pipeline.py
ODE Distillation roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py

Agentic Training

Agentic RL roll/pipeline/agentic/agentic_pipeline.py
Agent Runner roll/pipeline/agentic/agent_runner/base.py
Task Environments roll/pipeline/agentic/env

Distributed Runtime

Pipeline Base roll/pipeline/base_pipeline.py
Worker Cluster roll/distributed/executor/cluster.py
Distributed Worker roll/distributed/executor/worker.py
Batch Protocol roll/distributed/scheduler/protocol.py

Models and Data

Dataset Collation roll/datasets/collator.py
Model Providers roll/models/model_providers.py

Run State

Checkpoint Manager roll/utils/checkpoint_manager.py
Metrics roll/utils/metrics/metrics_manager.py

Other components

Training User

Connections 33

Training User → SFT: selects pipeline README.md
Training User → Preference Tuning: selects pipeline README.md
Training User → Distillation: selects pipeline README.md
Training User → Verifiable RL: selects pipeline README.md
Training User → Agentic RL: selects pipeline README.md
Training User → Diffusion Training: selects pipeline README.md
Pipeline Base → Worker Cluster: creates workers roll/pipeline/base_pipeline.py
Pipeline Base → Checkpoint Manager: uploads checkpoints roll/pipeline/base_pipeline.py
SFT → Pipeline Base: extends runtime roll/pipeline/sft/sft_pipeline.py
Preference Tuning → Pipeline Base: extends runtime roll/pipeline/dpo/dpo_pipeline.py
Distillation → Pipeline Base: extends runtime roll/pipeline/distill/distill_pipeline.py
Verifiable RL → Pipeline Base: extends runtime roll/pipeline/rlvr/rlvr_pipeline.py
Diffusion Training → Pipeline Base: extends runtime roll/pipeline/diffusion/diffusion_pipeline.py
DMD Distillation → Pipeline Base: extends runtime roll/pipeline/diffusion/dmd/dmd_pipeline.py
ODE Distillation → Pipeline Base: extends runtime roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py
Agentic RL → Pipeline Base: extends runtime roll/pipeline/agentic/agentic_pipeline.py
SFT → Dataset Collation: uses collation roll/pipeline/sft/sft_pipeline.py
Preference Tuning → Dataset Collation: uses collation roll/pipeline/dpo/dpo_pipeline.py
Distillation → Dataset Collation: uses collation roll/pipeline/distill/distill_pipeline.py
Verifiable RL → Dataset Collation: uses collation roll/pipeline/rlvr/rlvr_pipeline.py
Diffusion Training → Dataset Collation: uses collation roll/pipeline/diffusion/diffusion_pipeline.py
SFT → Model Providers: loads models roll/pipeline/sft/sft_pipeline.py
Preference Tuning → Model Providers: loads models roll/pipeline/dpo/dpo_pipeline.py
Distillation → Model Providers: loads models roll/pipeline/distill/distill_pipeline.py
Verifiable RL → Model Providers: loads models roll/pipeline/rlvr/rlvr_pipeline.py
Agentic RL → Model Providers: loads models roll/pipeline/agentic/agentic_pipeline.py
Diffusion Training → Model Providers: loads models roll/pipeline/diffusion/diffusion_pipeline.py
Worker Cluster → Distributed Worker: dispatches calls roll/distributed/executor/cluster.py
Distributed Worker → Batch Protocol: uses batch protocol roll/distributed/executor/worker.py
Agentic RL → Agent Runner: runs agents roll/pipeline/agentic/agentic_pipeline.py
Agentic RL → Task Environments: interacts with tasks README.md
DMD Distillation → Dataset Collation: uses dataset roll/pipeline/diffusion/dmd/dmd_pipeline.py
ODE Distillation → Dataset Collation: uses dataset roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py

More Python diagrams: pathwaycom/bdh · codegraphcontext/codegraphcontext · ruc-nlpir/flashrag · luigifreda/pyslam · lxf746/any-auto-register

Drawn by GitDiagram on September 30, 2026 from the repository's file tree, README and source files. Read as Markdown ·