# verl-project/verl-omni architecture

> Architecture diagram of the GitHub repository verl-project/verl-omni, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/verl-project/verl-omni
- Repository: https://github.com/verl-project/verl-omni

## Overview

VeRL-Omni is a reinforcement-learning post-training framework for diffusion, unified multimodal, and omni-modality generative models. Its principal workflow prepares training data, generates model rollouts, scores outputs with rewards, and updates actors through distributed trainers; diffusion also supports teacher scoring and rollout correction. Model-specific pipeline adapters connect common training and rollout machinery to supported models and algorithms. The README documents asynchronous rewards and selectable training backends. Sampled excerpts are partial, so the graph shows documented orchestration and only source-supported internal wiring, not every model-specific route.

Main entry points: [`verl_omni/trainer/diffusion/v1/trainer_sync.py`](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/v1/trainer_sync.py) (Diffusion trainer), [`verl_omni/trainer/omni/ray_omni_trainer.py`](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/omni/ray_omni_trainer.py) (Omni trainer), [`verl_omni/workers/engine_workers.py`](https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/engine_workers.py) (Actor workers), [`verl_omni/agent_loop/composite_agent_loop.py`](https://github.com/verl-project/verl-omni/blob/main/verl_omni/agent_loop/composite_agent_loop.py) (Agent loops)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training orchestration"]
  node_diffusion_entry["Diffusion entry"]
  node_omni_entry["Omni entry<br/>[main_omni.py]"]
  node_diffusion_trainer["Diffusion trainer<br/>[trainer_sync.py]"]
  node_async_trainer["Async diffusion"]
  node_omni_trainer["Omni trainer"]
  node_algorithms["RL algorithms<br/>[diffusion_algos.py]"]
end

subgraph group_generation["Rollout generation"]
  node_agent_loops["Agent loops"]
  node_rollout_worker["Rollout worker<br/>[base.py]"]
end

subgraph group_evaluation["Rewards and scoring"]
  node_reward_loop["Reward orchestration<br/>[reward_loop.py]"]
  node_reward_managers["Reward managers<br/>[multi.py]"]
  node_scorers["Reward scorers<br/>[unified_reward.py]"]
  node_teacher_manager["Teacher scoring<br/>[teacher_manager.py]"]
end

subgraph group_modelsdata["Data and model adapters"]
  node_datasets["RL datasets<br/>[rl_dataset.py]"]
  node_dpo_dataset["Offline DPO data"]
  node_model_adapters["Model adapters"]
end

subgraph group_runtime["Distributed runtime"]
  node_actor_workers["Actor workers<br/>[engine_workers.py]"]
  node_engines["Training engines"]
  node_weight_sync["Weight synchronization"]
end

node_researcher(("Training user"))
node_vllm["vLLM-Omni"]
node_scorer_service["Scorer service"]

node_researcher -->|"starts training"| node_diffusion_entry
node_researcher -->|"starts training"| node_omni_entry
node_diffusion_entry -->|"launches"| node_diffusion_trainer
node_diffusion_entry -->|"launches"| node_async_trainer
node_omni_entry -->|"launches"| node_omni_trainer
node_datasets -->|"provides batches"| node_diffusion_trainer
node_dpo_dataset -->|"provides pairs"| node_diffusion_trainer
node_diffusion_trainer -->|"requests rollouts"| node_agent_loops
node_async_trainer -->|"requests rollouts"| node_agent_loops
node_omni_trainer -->|"requests rollouts"| node_agent_loops
node_agent_loops -->|"dispatches requests"| node_rollout_worker
node_rollout_worker -->|"routes generation"| node_model_adapters
node_model_adapters -->|"generates outputs"| node_vllm
node_diffusion_trainer -->|"submits rollouts"| node_reward_loop
node_omni_trainer -->|"submits rollouts"| node_reward_loop
node_reward_loop -->|"dispatches scoring"| node_reward_managers
node_reward_managers -->|"invokes scorers"| node_scorers
node_scorers -.->|"requests scores"| node_scorer_service
node_diffusion_trainer -.->|"requests teacher scores"| node_teacher_manager
node_diffusion_trainer -->|"computes updates"| node_algorithms
node_omni_trainer -->|"updates actor"| node_actor_workers
node_diffusion_trainer -->|"updates actor"| node_actor_workers
node_actor_workers -->|"runs training"| node_engines
node_actor_workers -->|"sends weights"| node_weight_sync
node_weight_sync -->|"updates rollout weights"| node_rollout_worker

click node_diffusion_entry "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/main_diffusion_v1.py"
click node_omni_entry "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/main_omni.py"
click node_diffusion_trainer "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/v1/trainer_sync.py"
click node_async_trainer "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/v1/trainer_separate_async.py"
click node_omni_trainer "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/omni/ray_omni_trainer.py"
click node_algorithms "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/diffusion_algos.py"
click node_datasets "https://github.com/verl-project/verl-omni/blob/main/verl_omni/utils/dataset/rl_dataset.py"
click node_dpo_dataset "https://github.com/verl-project/verl-omni/blob/main/verl_omni/utils/dataset/offline_mllm_dpo_dataset.py"
click node_agent_loops "https://github.com/verl-project/verl-omni/blob/main/verl_omni/agent_loop/composite_agent_loop.py"
click node_rollout_worker "https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/rollout/base.py"
click node_model_adapters "https://github.com/verl-project/verl-omni/tree/main/verl_omni/pipelines"
click node_reward_loop "https://github.com/verl-project/verl-omni/blob/main/verl_omni/reward_loop/reward_loop.py"
click node_reward_managers "https://github.com/verl-project/verl-omni/blob/main/verl_omni/reward_loop/reward_manager/multi.py"
click node_scorers "https://github.com/verl-project/verl-omni/blob/main/verl_omni/utils/reward_score/unified_reward.py"
click node_teacher_manager "https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/teacher_manager.py"
click node_actor_workers "https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/engine_workers.py"
click node_engines "https://github.com/verl-project/verl-omni/tree/main/verl_omni/workers/engine"
click node_weight_sync "https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/checkpoint_engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_diffusion_entry,node_omni_entry,node_diffusion_trainer,node_async_trainer,node_omni_trainer,node_algorithms,node_researcher toneBlue
class node_agent_loops,node_rollout_worker toneAmber
class node_reward_loop,node_reward_managers,node_scorers,node_teacher_manager toneMint
class node_datasets,node_dpo_dataset,node_model_adapters toneRose
class node_actor_workers,node_engines,node_weight_sync,node_vllm,node_scorer_service toneIndigo
```

## Components

### Training orchestration

- [Diffusion entry](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/main_diffusion_v1.py): `verl_omni/trainer/main_diffusion_v1.py`
- [Omni entry](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/main_omni.py): `verl_omni/trainer/main_omni.py`
- [Diffusion trainer](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/v1/trainer_sync.py): `verl_omni/trainer/diffusion/v1/trainer_sync.py`
- [Async diffusion](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/v1/trainer_separate_async.py): `verl_omni/trainer/diffusion/v1/trainer_separate_async.py`
- [Omni trainer](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/omni/ray_omni_trainer.py): `verl_omni/trainer/omni/ray_omni_trainer.py`
- [RL algorithms](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/diffusion_algos.py): `verl_omni/trainer/diffusion/diffusion_algos.py`

### Rollout generation

- [Agent loops](https://github.com/verl-project/verl-omni/blob/main/verl_omni/agent_loop/composite_agent_loop.py): `verl_omni/agent_loop/composite_agent_loop.py`
- [Rollout worker](https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/rollout/base.py): `verl_omni/workers/rollout/base.py`

### Rewards and scoring

- [Reward orchestration](https://github.com/verl-project/verl-omni/blob/main/verl_omni/reward_loop/reward_loop.py): `verl_omni/reward_loop/reward_loop.py`
- [Reward managers](https://github.com/verl-project/verl-omni/blob/main/verl_omni/reward_loop/reward_manager/multi.py): `verl_omni/reward_loop/reward_manager/multi.py`
- [Reward scorers](https://github.com/verl-project/verl-omni/blob/main/verl_omni/utils/reward_score/unified_reward.py): `verl_omni/utils/reward_score/unified_reward.py`
- [Teacher scoring](https://github.com/verl-project/verl-omni/blob/main/verl_omni/trainer/diffusion/teacher_manager.py): `verl_omni/trainer/diffusion/teacher_manager.py`

### Data and model adapters

- [RL datasets](https://github.com/verl-project/verl-omni/blob/main/verl_omni/utils/dataset/rl_dataset.py): `verl_omni/utils/dataset/rl_dataset.py`
- [Offline DPO data](https://github.com/verl-project/verl-omni/blob/main/verl_omni/utils/dataset/offline_mllm_dpo_dataset.py): `verl_omni/utils/dataset/offline_mllm_dpo_dataset.py`
- [Model adapters](https://github.com/verl-project/verl-omni/tree/main/verl_omni/pipelines): `verl_omni/pipelines`

### Distributed runtime

- [Actor workers](https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/engine_workers.py): `verl_omni/workers/engine_workers.py`
- [Training engines](https://github.com/verl-project/verl-omni/tree/main/verl_omni/workers/engine): `verl_omni/workers/engine`
- [Weight synchronization](https://github.com/verl-project/verl-omni/blob/main/verl_omni/workers/checkpoint_engine.py): `verl_omni/workers/checkpoint_engine.py`

### Other components

- **Training user**
- **vLLM-Omni**
- **Scorer service**

## Connections

- Training user → Diffusion entry: starts training
- Training user → Omni entry: starts training
- Diffusion entry → Diffusion trainer: launches
- Diffusion entry → Async diffusion: launches
- Omni entry → Omni trainer: launches
- RL datasets → Diffusion trainer: provides batches
- Offline DPO data → Diffusion trainer: provides pairs
- Diffusion trainer → Agent loops: requests rollouts
- Async diffusion → Agent loops: requests rollouts
- Omni trainer → Agent loops: requests rollouts
- Agent loops → Rollout worker: dispatches requests
- Rollout worker → Model adapters: routes generation
- Model adapters → vLLM-Omni: generates outputs
- Diffusion trainer → Reward orchestration: submits rollouts
- Omni trainer → Reward orchestration: submits rollouts
- Reward orchestration → Reward managers: dispatches scoring
- Reward managers → Reward scorers: invokes scorers
- Reward scorers → Scorer service: requests scores
- Diffusion trainer → Teacher scoring: requests teacher scores
- Diffusion trainer → RL algorithms: computes updates
- Omni trainer → Actor workers: updates actor
- Diffusion trainer → Actor workers: updates actor
- Actor workers → Training engines: runs training
- Actor workers → Weight synchronization: sends weights
- Weight synchronization → Rollout worker: updates rollout weights