# alibaba/roll architecture

> Architecture diagram of the GitHub repository alibaba/roll, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba/roll
- Repository: https://github.com/alibaba/roll

## Overview

ROLL is a distributed library for large-scale LLM reinforcement-learning and post-training. Its principal workflow configures a training run, prepares data, then runs a selected pipeline across distributed workers to produce trained or aligned models. Major documented branches include SFT, preference optimization, distillation, RL with verifiable rewards, agentic interaction, and diffusion training. The graph includes shared model, scheduling, and checkpoint layers. Some pipeline-to-runtime links are documented or indexed, but unsampled implementation details are left unverified rather than inferred.

Main entry points: [`roll/pipeline/base_pipeline.py`](https://github.com/alibaba/roll/blob/main/roll/pipeline/base_pipeline.py) (Pipeline Base), [`roll/datasets/collator.py`](https://github.com/alibaba/roll/blob/main/roll/datasets/collator.py) (Dataset Collation), [`roll/models/model_providers.py`](https://github.com/alibaba/roll/blob/main/roll/models/model_providers.py) (Model Providers), [`roll/pipeline/agentic/agentic_pipeline.py`](https://github.com/alibaba/roll/blob/main/roll/pipeline/agentic/agentic_pipeline.py) (Agentic RL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipelines["Training Pipelines"]
  node_sft["SFT<br/>[sft_pipeline.py]"]
  node_dpo["Preference Tuning<br/>[dpo_pipeline.py]"]
  node_distill["Distillation"]
  node_rlvr["Verifiable RL<br/>[rlvr_pipeline.py]"]
  node_diffusion["Diffusion Training"]
  node_dmd["DMD Distillation<br/>[dmd_pipeline.py]"]
  node_ode["ODE Distillation"]
end

subgraph group_agentic["Agentic Training"]
  node_agenticpipe["Agentic RL"]
  node_agentrunner["Agent Runner<br/>[base.py]"]
  node_environment["Task Environments"]
end

subgraph group_runtime["Distributed Runtime"]
  node_base["Pipeline Base<br/>[base_pipeline.py]"]
  node_cluster["Worker Cluster<br/>[cluster.py]"]
  node_worker["Distributed Worker<br/>[worker.py]"]
  node_protocol["Batch Protocol<br/>[protocol.py]"]
end

subgraph group_modeldata["Models and Data"]
  node_datasets["Dataset Collation<br/>[collator.py]"]
  node_models["Model Providers<br/>[model_providers.py]"]
end

subgraph group_state["Run State"]
  node_checkpoints["Checkpoint Manager"]
  node_metrics["Metrics<br/>[metrics_manager.py]"]
end

node_engineer(("Training User"))

node_engineer -.->|"selects pipeline"| node_sft
node_engineer -.->|"selects pipeline"| node_dpo
node_engineer -.->|"selects pipeline"| node_distill
node_engineer -.->|"selects pipeline"| node_rlvr
node_engineer -.->|"selects pipeline"| node_agenticpipe
node_engineer -.->|"selects pipeline"| node_diffusion
node_base -->|"creates workers"| node_cluster
node_base -->|"uploads checkpoints"| node_checkpoints
node_sft -.->|"extends runtime"| node_base
node_dpo -.->|"extends runtime"| node_base
node_distill -.->|"extends runtime"| node_base
node_rlvr -.->|"extends runtime"| node_base
node_diffusion -.->|"extends runtime"| node_base
node_dmd -.->|"extends runtime"| node_base
node_ode -.->|"extends runtime"| node_base
node_agenticpipe -.->|"extends runtime"| node_base
node_sft -.->|"uses collation"| node_datasets
node_dpo -.->|"uses collation"| node_datasets
node_distill -.->|"uses collation"| node_datasets
node_rlvr -.->|"uses collation"| node_datasets
node_diffusion -.->|"uses collation"| node_datasets
node_sft -.->|"loads models"| node_models
node_dpo -.->|"loads models"| node_models
node_distill -.->|"loads models"| node_models
node_rlvr -.->|"loads models"| node_models
node_agenticpipe -.->|"loads models"| node_models
node_diffusion -.->|"loads models"| node_models
node_cluster -->|"dispatches calls"| node_worker
node_worker -.->|"uses batch protocol"| node_protocol
node_agenticpipe -.->|"runs agents"| node_agentrunner
node_agenticpipe -.->|"interacts with tasks"| node_environment
node_dmd -.->|"uses dataset"| node_datasets
node_ode -.->|"uses dataset"| node_datasets

click node_base "https://github.com/alibaba/roll/blob/main/roll/pipeline/base_pipeline.py"
click node_sft "https://github.com/alibaba/roll/blob/main/roll/pipeline/sft/sft_pipeline.py"
click node_dpo "https://github.com/alibaba/roll/blob/main/roll/pipeline/dpo/dpo_pipeline.py"
click node_distill "https://github.com/alibaba/roll/blob/main/roll/pipeline/distill/distill_pipeline.py"
click node_rlvr "https://github.com/alibaba/roll/blob/main/roll/pipeline/rlvr/rlvr_pipeline.py"
click node_diffusion "https://github.com/alibaba/roll/blob/main/roll/pipeline/diffusion/diffusion_pipeline.py"
click node_dmd "https://github.com/alibaba/roll/blob/main/roll/pipeline/diffusion/dmd/dmd_pipeline.py"
click node_ode "https://github.com/alibaba/roll/blob/main/roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py"
click node_agenticpipe "https://github.com/alibaba/roll/blob/main/roll/pipeline/agentic/agentic_pipeline.py"
click node_agentrunner "https://github.com/alibaba/roll/blob/main/roll/pipeline/agentic/agent_runner/base.py"
click node_environment "https://github.com/alibaba/roll/tree/main/roll/pipeline/agentic/env"
click node_datasets "https://github.com/alibaba/roll/blob/main/roll/datasets/collator.py"
click node_models "https://github.com/alibaba/roll/blob/main/roll/models/model_providers.py"
click node_cluster "https://github.com/alibaba/roll/blob/main/roll/distributed/executor/cluster.py"
click node_worker "https://github.com/alibaba/roll/blob/main/roll/distributed/executor/worker.py"
click node_protocol "https://github.com/alibaba/roll/blob/main/roll/distributed/scheduler/protocol.py"
click node_checkpoints "https://github.com/alibaba/roll/blob/main/roll/utils/checkpoint_manager.py"
click node_metrics "https://github.com/alibaba/roll/blob/main/roll/utils/metrics/metrics_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sft,node_dpo,node_distill,node_rlvr,node_diffusion,node_dmd,node_ode,node_engineer toneBlue
class node_agenticpipe,node_agentrunner,node_environment toneAmber
class node_base,node_cluster,node_worker,node_protocol toneMint
class node_datasets,node_models toneRose
class node_checkpoints,node_metrics toneIndigo
```

## Components

### Training Pipelines

- [SFT](https://github.com/alibaba/roll/blob/main/roll/pipeline/sft/sft_pipeline.py): `roll/pipeline/sft/sft_pipeline.py`
- [Preference Tuning](https://github.com/alibaba/roll/blob/main/roll/pipeline/dpo/dpo_pipeline.py): `roll/pipeline/dpo/dpo_pipeline.py`
- [Distillation](https://github.com/alibaba/roll/blob/main/roll/pipeline/distill/distill_pipeline.py): `roll/pipeline/distill/distill_pipeline.py`
- [Verifiable RL](https://github.com/alibaba/roll/blob/main/roll/pipeline/rlvr/rlvr_pipeline.py): `roll/pipeline/rlvr/rlvr_pipeline.py`
- [Diffusion Training](https://github.com/alibaba/roll/blob/main/roll/pipeline/diffusion/diffusion_pipeline.py): `roll/pipeline/diffusion/diffusion_pipeline.py`
- [DMD Distillation](https://github.com/alibaba/roll/blob/main/roll/pipeline/diffusion/dmd/dmd_pipeline.py): `roll/pipeline/diffusion/dmd/dmd_pipeline.py`
- [ODE Distillation](https://github.com/alibaba/roll/blob/main/roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py): `roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py`

### Agentic Training

- [Agentic RL](https://github.com/alibaba/roll/blob/main/roll/pipeline/agentic/agentic_pipeline.py): `roll/pipeline/agentic/agentic_pipeline.py`
- [Agent Runner](https://github.com/alibaba/roll/blob/main/roll/pipeline/agentic/agent_runner/base.py): `roll/pipeline/agentic/agent_runner/base.py`
- [Task Environments](https://github.com/alibaba/roll/tree/main/roll/pipeline/agentic/env): `roll/pipeline/agentic/env`

### Distributed Runtime

- [Pipeline Base](https://github.com/alibaba/roll/blob/main/roll/pipeline/base_pipeline.py): `roll/pipeline/base_pipeline.py`
- [Worker Cluster](https://github.com/alibaba/roll/blob/main/roll/distributed/executor/cluster.py): `roll/distributed/executor/cluster.py`
- [Distributed Worker](https://github.com/alibaba/roll/blob/main/roll/distributed/executor/worker.py): `roll/distributed/executor/worker.py`
- [Batch Protocol](https://github.com/alibaba/roll/blob/main/roll/distributed/scheduler/protocol.py): `roll/distributed/scheduler/protocol.py`

### Models and Data

- [Dataset Collation](https://github.com/alibaba/roll/blob/main/roll/datasets/collator.py): `roll/datasets/collator.py`
- [Model Providers](https://github.com/alibaba/roll/blob/main/roll/models/model_providers.py): `roll/models/model_providers.py`

### Run State

- [Checkpoint Manager](https://github.com/alibaba/roll/blob/main/roll/utils/checkpoint_manager.py): `roll/utils/checkpoint_manager.py`
- [Metrics](https://github.com/alibaba/roll/blob/main/roll/utils/metrics/metrics_manager.py): `roll/utils/metrics/metrics_manager.py`

### Other components

- **Training User**

## Connections

- Training User → SFT: selects pipeline (evidence: `README.md`)
- Training User → Preference Tuning: selects pipeline (evidence: `README.md`)
- Training User → Distillation: selects pipeline (evidence: `README.md`)
- Training User → Verifiable RL: selects pipeline (evidence: `README.md`)
- Training User → Agentic RL: selects pipeline (evidence: `README.md`)
- Training User → Diffusion Training: selects pipeline (evidence: `README.md`)
- Pipeline Base → Worker Cluster: creates workers (evidence: `roll/pipeline/base_pipeline.py`)
- Pipeline Base → Checkpoint Manager: uploads checkpoints (evidence: `roll/pipeline/base_pipeline.py`)
- SFT → Pipeline Base: extends runtime (evidence: `roll/pipeline/sft/sft_pipeline.py`)
- Preference Tuning → Pipeline Base: extends runtime (evidence: `roll/pipeline/dpo/dpo_pipeline.py`)
- Distillation → Pipeline Base: extends runtime (evidence: `roll/pipeline/distill/distill_pipeline.py`)
- Verifiable RL → Pipeline Base: extends runtime (evidence: `roll/pipeline/rlvr/rlvr_pipeline.py`)
- Diffusion Training → Pipeline Base: extends runtime (evidence: `roll/pipeline/diffusion/diffusion_pipeline.py`)
- DMD Distillation → Pipeline Base: extends runtime (evidence: `roll/pipeline/diffusion/dmd/dmd_pipeline.py`)
- ODE Distillation → Pipeline Base: extends runtime (evidence: `roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py`)
- Agentic RL → Pipeline Base: extends runtime (evidence: `roll/pipeline/agentic/agentic_pipeline.py`)
- SFT → Dataset Collation: uses collation (evidence: `roll/pipeline/sft/sft_pipeline.py`)
- Preference Tuning → Dataset Collation: uses collation (evidence: `roll/pipeline/dpo/dpo_pipeline.py`)
- Distillation → Dataset Collation: uses collation (evidence: `roll/pipeline/distill/distill_pipeline.py`)
- Verifiable RL → Dataset Collation: uses collation (evidence: `roll/pipeline/rlvr/rlvr_pipeline.py`)
- Diffusion Training → Dataset Collation: uses collation (evidence: `roll/pipeline/diffusion/diffusion_pipeline.py`)
- SFT → Model Providers: loads models (evidence: `roll/pipeline/sft/sft_pipeline.py`)
- Preference Tuning → Model Providers: loads models (evidence: `roll/pipeline/dpo/dpo_pipeline.py`)
- Distillation → Model Providers: loads models (evidence: `roll/pipeline/distill/distill_pipeline.py`)
- Verifiable RL → Model Providers: loads models (evidence: `roll/pipeline/rlvr/rlvr_pipeline.py`)
- Agentic RL → Model Providers: loads models (evidence: `roll/pipeline/agentic/agentic_pipeline.py`)
- Diffusion Training → Model Providers: loads models (evidence: `roll/pipeline/diffusion/diffusion_pipeline.py`)
- Worker Cluster → Distributed Worker: dispatches calls (evidence: `roll/distributed/executor/cluster.py`)
- Distributed Worker → Batch Protocol: uses batch protocol (evidence: `roll/distributed/executor/worker.py`)
- Agentic RL → Agent Runner: runs agents (evidence: `roll/pipeline/agentic/agentic_pipeline.py`)
- Agentic RL → Task Environments: interacts with tasks (evidence: `README.md`)
- DMD Distillation → Dataset Collation: uses dataset (evidence: `roll/pipeline/diffusion/dmd/dmd_pipeline.py`)
- ODE Distillation → Dataset Collation: uses dataset (evidence: `roll/pipeline/diffusion/ode_distillation/ode_distillation_pipeline.py`)