How xiaomimimo/verl works
This repository is XiaomiMiMo/verl: MiMo-focused agentic reinforcement-learning recipes built on the verl training library. The documented workflow is to launch training, generate agent trajectories in domain environments, judge task results, and use RL training to update models. The recipes cover code, cyber, general knowledge work, visual web development, and music; the underlying library supplies distributed workers, model engines, rollouts, reward handling, and checkpointing. Several recipe and trainer connections below are documented by the README, since their implementations were not sampled.
Main entry points: ray_trainer.py PPO trainer · agent.py Visual web tasks · engine_workers.py Engine workers · mimoagent_runner.py Code tasks
Components 24
RL training
- PPO trainer
verl/trainer/ppo/ray_trainer.py - PPO algorithms
verl/trainer/ppo/core_algos.py - Reward managers
verl/workers/reward_manager/registry.py
Domain recipes
- Code tasks
recipes/code/mimoagent_runner.py - Cyber tasks
recipes/arvo/agent_loop.py - General tasks
recipes/general/agent_loop.py - Visual web tasks
recipes/design/webdev/agent.py - Music tasks
recipes/design/music/scorer/pipeline.py - Code dataset
recipes/code/dataset.py - Web environment
recipes/design/webdev/environment.py
Model workers
- Engine workers
verl/workers/engine_workers.py - Engine configuration
verl/workers/config/engine.py - Inference rollouts
verl/workers/rollout/base.py - Weight synchronization
verl/checkpoint_engine/base.py
Distributed runtime
- Batch protocol
verl/protocol.py - Worker controller
verl/single_controller/base/decorator.py - Ray worker groups
verl/single_controller/ray/base.py - Checkpoint managers
verl/utils/checkpoint/checkpoint_manager.py - Device platforms
verl/plugin/platform/platform_manager.py - Device utilities
verl/utils/device.py - Runtime configuration
verl/utils/config.py
Task judging
- Visual grader
recipes/design/grader_service/src/design_grader/service/server.py - Music scoring
recipes/design/music/scorer/score.py
Other components
- Training operator
Connections 23
- Training operator → PPO trainer: launches
- PPO trainer → PPO algorithms: uses algorithms
- PPO trainer → Engine workers: trains with
- PPO trainer → Inference rollouts: collects trajectories
- PPO trainer → Reward managers: evaluates rewards
- PPO trainer → Weight synchronization: syncs weights
verl/checkpoint_engine/base.py - PPO trainer → Checkpoint managers: saves state
- PPO trainer → Worker controller: dispatches workers
- Worker controller → Ray worker groups: uses Ray groups
- Engine workers → Engine configuration: uses configuration
verl/workers/engine_workers.py - Engine workers → Batch protocol: exchanges batches
verl/workers/engine_workers.py - Device utilities → Device platforms: selects platform
verl/utils/device.py - Code tasks → Code dataset: uses dataset
- Code tasks → PPO trainer: runs within
- Cyber tasks → PPO trainer: runs within
- General tasks → PPO trainer: runs within
- Visual web tasks → PPO trainer: runs within
- Visual web tasks → Web environment: acts in
- Visual web tasks → Visual grader: requests grading
- Music tasks → Music scoring: scores compositions
recipes/design/music/scorer/pipeline.py - Music tasks → PPO trainer: trains within
- Reward managers → PPO trainer: returns reward
- Runtime configuration → Engine configuration: configures
verl/workers/config/engine.py