# hiyouga/easyr1 architecture

> Architecture diagram of the GitHub repository hiyouga/easyr1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hiyouga/easyr1
- Repository: https://github.com/hiyouga/easyr1

## Overview

##### Main Components
- Core Training Engine: Highlight the “verl/trainer” directory that houses the main training logic, configuration, metric computation, and integration with distributed training (e.g., “ray_trainer.py”).
- Model and Algorithm Management: Emphasize “verl/models” which contains customized model implementations (like “qwen2_vl.py”) and any monkey patching or transformer utilities.
- Worker Architecture: Under “verl/workers” and “verl/single_controller”, the framework implements different types of workers (actors, critics, rollout, reward, sharding managers) that handle the distributed RL training operations. This is critical in supporting scalable and parallel computing, especially using Ray and SPMD modes.
- Utility & Support Modules: “verl/utils” includes essential helper modules for checkpointing, logging, dataset management, tokenization, model utilities, and various custom math or torch functions.
- Configuration and Protocol: “verl/protocol.py” likely defines interfaces and protocols for interaction between different components.
- External Integration and Deployment: Files like Dockerfile(s), Makefile, and configuration examples (in “examples” and “scripts”) reveal how the system is containerized, deployed, and extended through external scripts.

##### Relationships and Interactions
- The Trainer (from the trainer module) is the central orchestrator that uses configurations and passes training instructions to the worker components (actors, critics, rollout managers).
- The Worker modules (organized with subdirectories for actor, critic, etc.) receive tasks from either a centralized “single_controller” or through Ray-based containers for distributed computing.
- Utility tools (e.g., for checkpointing and logging) are used by almost every component for recording training progress and handling failures.
- External scripts and configuration examples (within “examples” and “scripts”) interface with the training pipeline by setting runtime parameters and steering the training routines.

##### Architectural Patterns and Design Principles
- Layered Architecture: There’s a clear separation between core training logic, distributed worker management, and utilities. This modular approach favors extensibility.
- Distributed and Scalable Design: The usage of Ray in “verl/single_controller/ray” and worker submodules shows adoption of distributed computing patterns, crucial for scaling up across multiple nodes (as mentioned in the README regarding multi-node environments).
- Custom Abstractions and Protocols: The “protocol” files and base worker implementations indicate that the system leverages abstract interfaces to enable flexibility in swapping algorithms/components.
- Integration with External Services: Incorporating logging tools (Wandb, SwanLab, Mlflow, Tensorboard) and containerization through Docker are also important architectural elements.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Integration and Deployment
    subgraph "External Integration"
        Docker["Dockerfile"]
        DockerNightly["Dockerfile.nightly"]
        Make["Makefile"]
        ModelMerger["Model Merger Script"]
        Examples["Examples"]
    end

    %% Core Training Engine
    subgraph "Core Training Engine"
        TrainerMain["Main Trainer"]
        RayTrainer["Ray Trainer"]
        ConfigNode["Config File"]
    end

    %% Worker Architecture
    subgraph "Worker Architecture"
        subgraph "Distributed Workers"
            ActorWorker["Actor Worker"]
            CriticWorker["Critic Worker"]
            RolloutWorker["Rollout Worker"]
            ShardingManager["Sharding Manager"]
        end
        SingleController["Single Controller"]
        RaySupport["Ray Support"]
    end

    %% Model and Algorithm Management
    subgraph "Model and Algorithm Management"
        MonkeyPatch["Monkey Patch Utils"]
        QwenModel["Qwen2_VL Model"]
    end

    %% Utility & Support Modules
    subgraph "Utility & Support Modules"
        Utils["Checkpoint, Logging, Tokenizer, etc."]
    end

    %% Configuration and Protocol
    subgraph "Configuration and Protocol"
        Protocol["Protocol Interface"]
    end

    %% External Integration to Trainer Engine
    Docker -->|"deploys"| TrainerMain
    DockerNightly -->|"deploys"| TrainerMain
    Make -->|"deploys"| TrainerMain
    ModelMerger -->|"deploys"| TrainerMain
    Examples -->|"deploys"| TrainerMain

    %% Communication between Trainer Engine and Worker Architecture
    TrainerMain -->|"dispatches"| ActorWorker
    TrainerMain -->|"dispatches"| CriticWorker
    TrainerMain -->|"dispatches"| RolloutWorker
    TrainerMain -->|"dispatches"| ShardingManager
    TrainerMain -->|"coordinates"| SingleController
    RayTrainer -->|"uses"| RaySupport

    ActorWorker -->|"reports"| TrainerMain
    CriticWorker -->|"reports"| TrainerMain
    RolloutWorker -->|"reports"| TrainerMain
    ShardingManager -->|"reports"| TrainerMain
    SingleController -->|"reports"| TrainerMain

    %% Interaction with Model and Utility Modules
    TrainerMain -->|"integrates"| MonkeyPatch
    TrainerMain -->|"integrates"| QwenModel

    TrainerMain -->|"logs"| Utils
    Utils -->|"feedback"| TrainerMain

    %% Interaction with Protocol Interface
    TrainerMain -->|"communicates"| Protocol
    ActorWorker -->|"communicates"| Protocol
    CriticWorker -->|"communicates"| Protocol
    RolloutWorker -->|"communicates"| Protocol
    SingleController -->|"communicates"| Protocol

    %% Internal Trainer Engine Communication
    TrainerMain -->|"spawns"| RayTrainer
    RayTrainer -->|"feedback"| TrainerMain

    %% Click Events
    click TrainerMain "https://github.com/hiyouga/easyr1/blob/main/verl/trainer/main.py"
    click RayTrainer "https://github.com/hiyouga/easyr1/blob/main/verl/trainer/ray_trainer.py"
    click ConfigNode "https://github.com/hiyouga/easyr1/blob/main/config.py"
    click MonkeyPatch "https://github.com/hiyouga/easyr1/blob/main/verl/models/monkey_patch.py"
    click QwenModel "https://github.com/hiyouga/easyr1/blob/main/verl/models/transformers/qwen2_vl.py"
    click ActorWorker "https://github.com/hiyouga/easyr1/tree/main/verl/workers"
    click CriticWorker "https://github.com/hiyouga/easyr1/tree/main/verl/workers"
    click RolloutWorker "https://github.com/hiyouga/easyr1/tree/main/verl/workers"
    click ShardingManager "https://github.com/hiyouga/easyr1/tree/main/verl/workers"
    click SingleController "https://github.com/hiyouga/easyr1/tree/main/verl/single_controller"
    click RaySupport "https://github.com/hiyouga/easyr1/tree/main/verl/single_controller/ray"
    click Utils "https://github.com/hiyouga/easyr1/tree/main/verl/utils"
    click Protocol "https://github.com/hiyouga/easyr1/blob/main/verl/protocol.py"
    click Docker "https://github.com/hiyouga/easyr1/tree/main/Dockerfile"
    click DockerNightly "https://github.com/hiyouga/easyr1/blob/main/Dockerfile.nightly"
    click Make "https://github.com/hiyouga/easyr1/tree/main/Makefile"
    click ModelMerger "https://github.com/hiyouga/easyr1/blob/main/scripts/model_merger.py"
    click Examples "https://github.com/hiyouga/easyr1/tree/main/examples"

    %% Styles
    classDef core fill:#ff9999,stroke:#333,stroke-width:2px;
    classDef worker fill:#ffcc99,stroke:#333,stroke-width:2px;
    classDef model fill:#99ccff,stroke:#333,stroke-width:2px;
    classDef utility fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef protocol fill:#e6ccff,stroke:#333,stroke-width:2px;
    classDef external fill:#cccccc,stroke:#333,stroke-width:2px;

    class TrainerMain,RayTrainer,ConfigNode core;
    class ActorWorker,CriticWorker,RolloutWorker,ShardingManager,SingleController,RaySupport worker;
    class MonkeyPatch,QwenModel model;
    class Utils utility;
    class Protocol protocol;
    class Docker,DockerNightly,Make,ModelMerger,Examples external;
```
