# xiaomimimo/verl architecture

> Architecture diagram of the GitHub repository xiaomimimo/verl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaomimimo/verl
- Repository: https://github.com/xiaomimimo/verl

## Overview

This repository is XiaomiMiMo/verl: MiMo-focused agentic reinforcement-learning recipes built on the verl training library. The documented workflow is to launch training, generate agent trajectories in domain environments, judge task results, and use RL training to update models. The recipes cover code, cyber, general knowledge work, visual web development, and music; the underlying library supplies distributed workers, model engines, rollouts, reward handling, and checkpointing. Several recipe and trainer connections below are documented by the README, since their implementations were not sampled.

Main entry points: [`verl/trainer/ppo/ray_trainer.py`](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/trainer/ppo/ray_trainer.py) (PPO trainer), [`recipes/design/webdev/agent.py`](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/webdev/agent.py) (Visual web tasks), [`verl/workers/engine_workers.py`](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/engine_workers.py) (Engine workers), [`recipes/code/mimoagent_runner.py`](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/code/mimoagent_runner.py) (Code tasks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["RL training"]
  node_trainer["PPO trainer<br/>[ray_trainer.py]"]
  node_ppo["PPO algorithms<br/>[core_algos.py]"]
  node_reward["Reward managers<br/>[registry.py]"]
end

subgraph group_recipes["Domain recipes"]
  node_code["Code tasks"]
  node_cyber["Cyber tasks<br/>[agent_loop.py]"]
  node_general["General tasks<br/>[agent_loop.py]"]
  node_visual["Visual web tasks<br/>[agent.py]"]
  node_music["Music tasks<br/>[pipeline.py]"]
  node_code_data["Code dataset<br/>[dataset.py]"]
  node_visual_env["Web environment<br/>[environment.py]"]
end

subgraph group_workers["Model workers"]
  node_engine_workers["Engine workers<br/>[engine_workers.py]"]
  node_engine_config["Engine configuration<br/>[engine.py]"]
  node_rollouts["Inference rollouts<br/>[base.py]"]
  node_checkpoint["Weight synchronization<br/>[base.py]"]
end

subgraph group_runtime["Distributed runtime"]
  node_protocol["Batch protocol<br/>[protocol.py]"]
  node_controller["Worker controller<br/>[decorator.py]"]
  node_worker_group["Ray worker groups<br/>[base.py]"]
  node_checkpoint_store["Checkpoint managers"]
  node_platform["Device platforms"]
  node_device["Device utilities<br/>[device.py]"]
  node_config["Runtime configuration<br/>[config.py]"]
end

subgraph group_judging["Task judging"]
  node_visual_grader["Visual grader<br/>[server.py]"]
  node_music_score["Music scoring<br/>[score.py]"]
end

node_operator(("Training operator"))

node_operator -.->|"launches"| node_trainer
node_trainer -.->|"uses algorithms"| node_ppo
node_trainer -.->|"trains with"| node_engine_workers
node_trainer -.->|"collects trajectories"| node_rollouts
node_trainer -.->|"evaluates rewards"| node_reward
node_trainer -->|"syncs weights"| node_checkpoint
node_trainer -.->|"saves state"| node_checkpoint_store
node_trainer -.->|"dispatches workers"| node_controller
node_controller -->|"uses Ray groups"| node_worker_group
node_engine_workers -->|"uses configuration"| node_engine_config
node_engine_workers -->|"exchanges batches"| node_protocol
node_device -->|"selects platform"| node_platform
node_code -.->|"uses dataset"| node_code_data
node_code -.->|"runs within"| node_trainer
node_cyber -->|"runs within"| node_trainer
node_general -->|"runs within"| node_trainer
node_visual -->|"runs within"| node_trainer
node_visual -.->|"acts in"| node_visual_env
node_visual -.->|"requests grading"| node_visual_grader
node_music -->|"scores compositions"| node_music_score
node_music -.->|"trains within"| node_trainer
node_reward -.->|"returns reward"| node_trainer
node_config -.->|"configures"| node_engine_config

click node_trainer "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/trainer/ppo/ray_trainer.py"
click node_ppo "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/trainer/ppo/core_algos.py"
click node_protocol "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/protocol.py"
click node_controller "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/single_controller/base/decorator.py"
click node_worker_group "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/single_controller/ray/base.py"
click node_engine_workers "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/engine_workers.py"
click node_engine_config "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/config/engine.py"
click node_rollouts "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/rollout/base.py"
click node_checkpoint "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/checkpoint_engine/base.py"
click node_checkpoint_store "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/utils/checkpoint/checkpoint_manager.py"
click node_code "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/code/mimoagent_runner.py"
click node_cyber "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/arvo/agent_loop.py"
click node_general "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/general/agent_loop.py"
click node_visual "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/webdev/agent.py"
click node_music "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/music/scorer/pipeline.py"
click node_code_data "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/code/dataset.py"
click node_visual_env "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/webdev/environment.py"
click node_visual_grader "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/grader_service/src/design_grader/service/server.py"
click node_music_score "https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/music/scorer/score.py"
click node_reward "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/reward_manager/registry.py"
click node_platform "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/plugin/platform/platform_manager.py"
click node_device "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/utils/device.py"
click node_config "https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/utils/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_trainer,node_ppo,node_reward toneBlue
class node_code,node_cyber,node_general,node_visual,node_music,node_code_data,node_visual_env toneAmber
class node_engine_workers,node_engine_config,node_rollouts,node_checkpoint toneMint
class node_protocol,node_controller,node_worker_group,node_checkpoint_store,node_platform,node_device,node_config toneRose
class node_visual_grader,node_music_score,node_operator toneIndigo
```

## Components

### RL training

- [PPO trainer](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/trainer/ppo/ray_trainer.py): `verl/trainer/ppo/ray_trainer.py`
- [PPO algorithms](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/trainer/ppo/core_algos.py): `verl/trainer/ppo/core_algos.py`
- [Reward managers](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/reward_manager/registry.py): `verl/workers/reward_manager/registry.py`

### Domain recipes

- [Code tasks](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/code/mimoagent_runner.py): `recipes/code/mimoagent_runner.py`
- [Cyber tasks](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/arvo/agent_loop.py): `recipes/arvo/agent_loop.py`
- [General tasks](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/general/agent_loop.py): `recipes/general/agent_loop.py`
- [Visual web tasks](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/webdev/agent.py): `recipes/design/webdev/agent.py`
- [Music tasks](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/music/scorer/pipeline.py): `recipes/design/music/scorer/pipeline.py`
- [Code dataset](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/code/dataset.py): `recipes/code/dataset.py`
- [Web environment](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/webdev/environment.py): `recipes/design/webdev/environment.py`

### Model workers

- [Engine workers](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/engine_workers.py): `verl/workers/engine_workers.py`
- [Engine configuration](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/config/engine.py): `verl/workers/config/engine.py`
- [Inference rollouts](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/workers/rollout/base.py): `verl/workers/rollout/base.py`
- [Weight synchronization](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/checkpoint_engine/base.py): `verl/checkpoint_engine/base.py`

### Distributed runtime

- [Batch protocol](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/protocol.py): `verl/protocol.py`
- [Worker controller](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/single_controller/base/decorator.py): `verl/single_controller/base/decorator.py`
- [Ray worker groups](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/single_controller/ray/base.py): `verl/single_controller/ray/base.py`
- [Checkpoint managers](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/utils/checkpoint/checkpoint_manager.py): `verl/utils/checkpoint/checkpoint_manager.py`
- [Device platforms](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/plugin/platform/platform_manager.py): `verl/plugin/platform/platform_manager.py`
- [Device utilities](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/utils/device.py): `verl/utils/device.py`
- [Runtime configuration](https://github.com/xiaomimimo/verl/blob/mimo-oss/verl/utils/config.py): `verl/utils/config.py`

### Task judging

- [Visual grader](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/grader_service/src/design_grader/service/server.py): `recipes/design/grader_service/src/design_grader/service/server.py`
- [Music scoring](https://github.com/xiaomimimo/verl/blob/mimo-oss/recipes/design/music/scorer/score.py): `recipes/design/music/scorer/score.py`

### Other components

- **Training operator**

## Connections

- Training operator → PPO trainer: launches
- PPO trainer → PPO algorithms: uses algorithms
- PPO trainer → Engine workers: trains with
- PPO trainer → Inference rollouts: collects trajectories
- PPO trainer → Reward managers: evaluates rewards
- PPO trainer → Weight synchronization: syncs weights (evidence: `verl/checkpoint_engine/base.py`)
- PPO trainer → Checkpoint managers: saves state
- PPO trainer → Worker controller: dispatches workers
- Worker controller → Ray worker groups: uses Ray groups
- Engine workers → Engine configuration: uses configuration (evidence: `verl/workers/engine_workers.py`)
- Engine workers → Batch protocol: exchanges batches (evidence: `verl/workers/engine_workers.py`)
- Device utilities → Device platforms: selects platform (evidence: `verl/utils/device.py`)
- Code tasks → Code dataset: uses dataset
- Code tasks → PPO trainer: runs within
- Cyber tasks → PPO trainer: runs within
- General tasks → PPO trainer: runs within
- Visual web tasks → PPO trainer: runs within
- Visual web tasks → Web environment: acts in
- Visual web tasks → Visual grader: requests grading
- Music tasks → Music scoring: scores compositions (evidence: `recipes/design/music/scorer/pipeline.py`)
- Music tasks → PPO trainer: trains within
- Reward managers → PPO trainer: returns reward
- Runtime configuration → Engine configuration: configures (evidence: `verl/workers/config/engine.py`)
