# xiyang-aads-lilly/roll architecture

> Architecture diagram of the GitHub repository xiyang-aads-lilly/roll, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiyang-aads-lilly/roll
- Repository: https://github.com/xiyang-aads-lilly/roll

## Overview

ROLL is a distributed library for training and aligning large language and multimodal models. Users configure a training pipeline, which prepares data and coordinates Ray-distributed model workers; depending on the pipeline, it performs supervised or preference training, generates rollouts and scores rewards, distills from a teacher, or trains diffusion reward models. Shared runtime services manage resources, model updates, checkpoints, and metrics. The README documents further model and hardware integrations; sampled excerpts confirm only some of their wiring, so uncertain links are omitted.

Main entry points: [`roll/pipeline/base_pipeline.py`](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/base_pipeline.py) (Pipeline base), [`roll/pipeline/rlvr/rlvr_pipeline.py`](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/rlvr/rlvr_pipeline.py) (RLVR), [`roll/pipeline/distill/distill_pipeline.py`](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/distill_pipeline.py) (Distillation), [`roll/distributed/executor/cluster.py`](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/cluster.py) (Worker clusters)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipelines["Training pipelines"]
  node_sft["SFT<br/>[sft_pipeline.py]"]
  node_dpo["DPO<br/>[dpo_pipeline.py]"]
  node_rlvr["RLVR<br/>[rlvr_pipeline.py]"]
  node_rlvr_vlm["RLVR VLM"]
  node_distill["Distillation"]
  node_distill_vlm["VLM distillation"]
  node_dpo_pipeline["DPO pipeline<br/>[dpo_pipeline.py]"]
  node_reward_fl["Diffusion reward"]
end

subgraph group_data["Data preparation"]
  node_dataset["Dataset loading<br/>[loader.py]"]
  node_collator["Batch collation<br/>[collator.py]"]
end

subgraph group_runtime["Distributed runtime"]
  node_base["Pipeline base<br/>[base_pipeline.py]"]
  node_scheduler["Rollout scheduling"]
  node_reward_scheduler["Reward scheduling"]
  node_cluster["Worker clusters<br/>[cluster.py]"]
  node_model_update["Model updates"]
  node_worker["Worker execution<br/>[worker.py]"]
  node_checkpoint["Checkpoint management"]
  node_tracker["Metrics tracking<br/>[tracking.py]"]
end

subgraph group_workers["Model workers"]
  node_strategy["Execution strategies<br/>[factory.py]"]
  node_logits["Logits transfer"]
end

subgraph group_agentic["Agentic interaction"]
  node_agentic_pipeline["Agentic training"]
  node_agentic_rollout["Agentic rollout"]
  node_env_manager["Environment manager"]
  node_tools["Agent tools<br/>[registration.py]"]
end

node_operator(("Training user"))
node_api(("Judge API"))

node_operator -->|"configures"| node_sft
node_operator -->|"configures"| node_dpo
node_operator -->|"configures"| node_rlvr
node_operator -->|"configures"| node_agentic_pipeline
node_operator -->|"configures"| node_distill
node_operator -->|"configures"| node_reward_fl
node_sft -->|"inherits"| node_base
node_dpo -->|"inherits"| node_base
node_rlvr -->|"inherits"| node_base
node_rlvr_vlm -->|"inherits"| node_base
node_distill -->|"inherits"| node_base
node_distill_vlm -->|"inherits"| node_base
node_agentic_pipeline -->|"inherits"| node_base
node_reward_fl -->|"inherits"| node_base
node_sft -->|"loads data"| node_dataset
node_sft -->|"batches data"| node_collator
node_dpo -->|"loads data"| node_dataset
node_distill -->|"loads data"| node_dataset
node_distill_vlm -->|"loads data"| node_dataset
node_reward_fl -->|"loads data"| node_dataset
node_rlvr -->|"initializes workers"| node_cluster
node_rlvr -->|"dispatches generation"| node_scheduler
node_rlvr -->|"schedules rewards"| node_reward_scheduler
node_rlvr_vlm -->|"dispatches generation"| node_scheduler
node_agentic_pipeline -->|"initializes workers"| node_cluster
node_agentic_pipeline -->|"coordinates rollout"| node_agentic_rollout
node_agentic_rollout -->|"manages episodes"| node_env_manager
node_env_manager -->|"dispatches tools"| node_tools
node_agentic_rollout -->|"schedules generation"| node_scheduler
node_distill -->|"initializes workers"| node_cluster
node_distill -->|"transfers logits"| node_logits
node_distill_vlm -->|"transfers logits"| node_logits
node_reward_fl -->|"initializes workers"| node_cluster
node_cluster -->|"dispatches work"| node_worker
node_worker -->|"selects strategy"| node_strategy
node_base -->|"updates models"| node_model_update
node_base -->|"saves checkpoints"| node_checkpoint
node_base -->|"logs metrics"| node_tracker
node_api -.->|"scores responses"| node_rlvr

click node_base "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/base_pipeline.py"
click node_sft "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/sft/sft_pipeline.py"
click node_dpo "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/dpo/dpo_pipeline.py"
click node_rlvr "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/rlvr/rlvr_pipeline.py"
click node_rlvr_vlm "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/rlvr/rlvr_vlm_pipeline.py"
click node_distill "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/distill_pipeline.py"
click node_distill_vlm "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/distill_vlm_pipeline.py"
click node_dpo_pipeline "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/dpo/dpo_pipeline.py"
click node_reward_fl "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/diffusion/reward_fl/reward_fl_pipeline.py"
click node_agentic_pipeline "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/agentic_pipeline.py"
click node_agentic_rollout "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/agentic_rollout_pipeline.py"
click node_env_manager "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/env_manager/base_env_manager.py"
click node_tools "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/tools/registration.py"
click node_dataset "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/datasets/loader.py"
click node_collator "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/datasets/collator.py"
click node_scheduler "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/scheduler/generate_scheduler.py"
click node_reward_scheduler "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/scheduler/reward_scheduler.py"
click node_cluster "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/cluster.py"
click node_model_update "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/model_update_group.py"
click node_strategy "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/strategy/factory.py"
click node_worker "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/worker.py"
click node_logits "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/logits_transfer_group.py"
click node_checkpoint "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/utils/checkpoint_manager.py"
click node_tracker "https://github.com/xiyang-aads-lilly/roll/blob/main/roll/utils/tracking.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sft,node_dpo,node_rlvr,node_rlvr_vlm,node_distill,node_distill_vlm,node_dpo_pipeline,node_reward_fl,node_operator toneBlue
class node_dataset,node_collator toneAmber
class node_base,node_scheduler,node_reward_scheduler,node_cluster,node_model_update,node_worker,node_checkpoint,node_tracker,node_api toneMint
class node_strategy,node_logits toneRose
class node_agentic_pipeline,node_agentic_rollout,node_env_manager,node_tools toneIndigo
```

## Components

### Training pipelines

- [SFT](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/sft/sft_pipeline.py): `roll/pipeline/sft/sft_pipeline.py`
- [DPO](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/dpo/dpo_pipeline.py): `roll/pipeline/dpo/dpo_pipeline.py`
- [RLVR](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/rlvr/rlvr_pipeline.py): `roll/pipeline/rlvr/rlvr_pipeline.py`
- [RLVR VLM](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/rlvr/rlvr_vlm_pipeline.py): `roll/pipeline/rlvr/rlvr_vlm_pipeline.py`
- [Distillation](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/distill_pipeline.py): `roll/pipeline/distill/distill_pipeline.py`
- [VLM distillation](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/distill_vlm_pipeline.py): `roll/pipeline/distill/distill_vlm_pipeline.py`
- [DPO pipeline](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/dpo/dpo_pipeline.py): `roll/pipeline/dpo/dpo_pipeline.py`
- [Diffusion reward](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/diffusion/reward_fl/reward_fl_pipeline.py): `roll/pipeline/diffusion/reward_fl/reward_fl_pipeline.py`

### Data preparation

- [Dataset loading](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/datasets/loader.py): `roll/datasets/loader.py`
- [Batch collation](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/datasets/collator.py): `roll/datasets/collator.py`

### Distributed runtime

- [Pipeline base](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/base_pipeline.py): `roll/pipeline/base_pipeline.py`
- [Rollout scheduling](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/scheduler/generate_scheduler.py): `roll/distributed/scheduler/generate_scheduler.py`
- [Reward scheduling](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/scheduler/reward_scheduler.py): `roll/distributed/scheduler/reward_scheduler.py`
- [Worker clusters](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/cluster.py): `roll/distributed/executor/cluster.py`
- [Model updates](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/model_update_group.py): `roll/distributed/executor/model_update_group.py`
- [Worker execution](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/executor/worker.py): `roll/distributed/executor/worker.py`
- [Checkpoint management](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/utils/checkpoint_manager.py): `roll/utils/checkpoint_manager.py`
- [Metrics tracking](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/utils/tracking.py): `roll/utils/tracking.py`

### Model workers

- [Execution strategies](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/distributed/strategy/factory.py): `roll/distributed/strategy/factory.py`
- [Logits transfer](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/distill/logits_transfer_group.py): `roll/pipeline/distill/logits_transfer_group.py`

### Agentic interaction

- [Agentic training](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/agentic_pipeline.py): `roll/pipeline/agentic/agentic_pipeline.py`
- [Agentic rollout](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/agentic_rollout_pipeline.py): `roll/pipeline/agentic/agentic_rollout_pipeline.py`
- [Environment manager](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/env_manager/base_env_manager.py): `roll/pipeline/agentic/env_manager/base_env_manager.py`
- [Agent tools](https://github.com/xiyang-aads-lilly/roll/blob/main/roll/pipeline/agentic/tools/registration.py): `roll/pipeline/agentic/tools/registration.py`

### Other components

- **Training user**
- **Judge API**

## Connections

- Training user → SFT: configures
- Training user → DPO: configures
- Training user → RLVR: configures
- Training user → Agentic training: configures
- Training user → Distillation: configures
- Training user → Diffusion reward: configures
- SFT → Pipeline base: inherits
- DPO → Pipeline base: inherits
- RLVR → Pipeline base: inherits
- RLVR VLM → Pipeline base: inherits
- Distillation → Pipeline base: inherits
- VLM distillation → Pipeline base: inherits
- Agentic training → Pipeline base: inherits
- Diffusion reward → Pipeline base: inherits
- SFT → Dataset loading: loads data
- SFT → Batch collation: batches data
- DPO → Dataset loading: loads data
- Distillation → Dataset loading: loads data
- VLM distillation → Dataset loading: loads data
- Diffusion reward → Dataset loading: loads data
- RLVR → Worker clusters: initializes workers
- RLVR → Rollout scheduling: dispatches generation
- RLVR → Reward scheduling: schedules rewards
- RLVR VLM → Rollout scheduling: dispatches generation
- Agentic training → Worker clusters: initializes workers
- Agentic training → Agentic rollout: coordinates rollout
- Agentic rollout → Environment manager: manages episodes
- Environment manager → Agent tools: dispatches tools
- Agentic rollout → Rollout scheduling: schedules generation
- Distillation → Worker clusters: initializes workers
- Distillation → Logits transfer: transfers logits
- VLM distillation → Logits transfer: transfers logits
- Diffusion reward → Worker clusters: initializes workers
- Worker clusters → Worker execution: dispatches work
- Worker execution → Execution strategies: selects strategy
- Pipeline base → Model updates: updates models
- Pipeline base → Checkpoint management: saves checkpoints
- Pipeline base → Metrics tracking: logs metrics
- Judge API → RLVR: scores responses