How xiyang-aads-lilly/roll works
ROLL is a distributed library for training and aligning large language and multimodal models. Users configure a training pipeline, which prepares data and coordinates Ray-distributed model workers; depending on the pipeline, it performs supervised or preference training, generates rollouts and scores rewards, distills from a teacher, or trains diffusion reward models. Shared runtime services manage resources, model updates, checkpoints, and metrics. The README documents further model and hardware integrations; sampled excerpts confirm only some of their wiring, so uncertain links are omitted.
Main entry points: base_pipeline.py Pipeline base · rlvr_pipeline.py RLVR · distill_pipeline.py Distillation · cluster.py Worker clusters
Components 26
Training pipelines
- SFT
roll/pipeline/sft/sft_pipeline.py - DPO
roll/pipeline/dpo/dpo_pipeline.py - RLVR
roll/pipeline/rlvr/rlvr_pipeline.py - RLVR VLM
roll/pipeline/rlvr/rlvr_vlm_pipeline.py - Distillation
roll/pipeline/distill/distill_pipeline.py - VLM distillation
roll/pipeline/distill/distill_vlm_pipeline.py - DPO pipeline
roll/pipeline/dpo/dpo_pipeline.py - Diffusion reward
roll/pipeline/diffusion/reward_fl/reward_fl_pipeline.py
Data preparation
- Dataset loading
roll/datasets/loader.py - Batch collation
roll/datasets/collator.py
Distributed runtime
- Pipeline base
roll/pipeline/base_pipeline.py - Rollout scheduling
roll/distributed/scheduler/generate_scheduler.py - Reward scheduling
roll/distributed/scheduler/reward_scheduler.py - Worker clusters
roll/distributed/executor/cluster.py - Model updates
roll/distributed/executor/model_update_group.py - Worker execution
roll/distributed/executor/worker.py - Checkpoint management
roll/utils/checkpoint_manager.py - Metrics tracking
roll/utils/tracking.py
Model workers
- Execution strategies
roll/distributed/strategy/factory.py - Logits transfer
roll/pipeline/distill/logits_transfer_group.py
Agentic interaction
- Agentic training
roll/pipeline/agentic/agentic_pipeline.py - Agentic rollout
roll/pipeline/agentic/agentic_rollout_pipeline.py - Environment manager
roll/pipeline/agentic/env_manager/base_env_manager.py - Agent tools
roll/pipeline/agentic/tools/registration.py
Other components
- Training user
- Judge API
Connections 39
- Training user → SFT: configures
- Training user → DPO: configures
- Training user → RLVR: configures
- Training user → Agentic training: configures
- Training user → Distillation: configures
- Training user → Diffusion reward: configures
- SFT → Pipeline base: inherits
- DPO → Pipeline base: inherits
- RLVR → Pipeline base: inherits
- RLVR VLM → Pipeline base: inherits
- Distillation → Pipeline base: inherits
- VLM distillation → Pipeline base: inherits
- Agentic training → Pipeline base: inherits
- Diffusion reward → Pipeline base: inherits
- SFT → Dataset loading: loads data
- SFT → Batch collation: batches data
- DPO → Dataset loading: loads data
- Distillation → Dataset loading: loads data
- VLM distillation → Dataset loading: loads data
- Diffusion reward → Dataset loading: loads data
- RLVR → Worker clusters: initializes workers
- RLVR → Rollout scheduling: dispatches generation
- RLVR → Reward scheduling: schedules rewards
- RLVR VLM → Rollout scheduling: dispatches generation
- Agentic training → Worker clusters: initializes workers
- Agentic training → Agentic rollout: coordinates rollout
- Agentic rollout → Environment manager: manages episodes
- Environment manager → Agent tools: dispatches tools
- Agentic rollout → Rollout scheduling: schedules generation
- Distillation → Worker clusters: initializes workers
- Distillation → Logits transfer: transfers logits
- VLM distillation → Logits transfer: transfers logits
- Diffusion reward → Worker clusters: initializes workers
- Worker clusters → Worker execution: dispatches work
- Worker execution → Execution strategies: selects strategy
- Pipeline base → Model updates: updates models
- Pipeline base → Checkpoint management: saves checkpoints
- Pipeline base → Metrics tracking: logs metrics
- Judge API → RLVR: scores responses