# ascend/mindspeed-rl architecture

> Architecture diagram of the GitHub repository ascend/mindspeed-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ascend/mindspeed-rl
- Repository: https://github.com/ascend/mindspeed-rl

## Overview

##### Main Components
- User Interface / CLI Layer:
– The “cli” directory contains scripts for conversion, inference, preprocessing, and training for different algorithms.
- Configuration & Data Management:
– The “configs” directory holds various configuration files for models, datasets, environments, and training settings.
– Include a component for “Configuration Manager” that loads and validates settings (also note the presence of config validation code in mindspeed_rl/config_cls).
- Core RL Engine Library (mindspeed_rl):
– This is the central package, which is subdivided into:
▪ “datasets” – Data handling, preprocessing, formatting, and sampling.
▪ “models” – Model definitions including actors, rollouts, loss functions, inference engines, and reference models.
▪ “trainer” – Trainer modules for different algorithms (GRPO, SFT, ORM) along with distributed training utilities.
▪ “workers” – Distributed worker components including schedulers and resharding modules.
▪ “utils” – Auxiliary modules for logging, computing, optimizer support, and other functions.
- Distributed and Parallel Training:
– Integration with Ray and vllm for asynchronous scheduling and parallel state management. This can be depicted as a “Distributed Scheduler / Worker Manager” that interacts with the trainers and workers.
- External Dependencies & Ecosystem:
– The README and configuration files indicate tight integration with Ascend NPU drivers, toolkit, and accelerator libraries (e.g., megatron weight loaders). Represent these as “External System Dependencies” or an “Ascend Ecosystem” block that communicates with core training processes.
- Documentation & Examples:

##### Relationships and Interactions
- Data Flow:
– Show how user commands from the CLI trigger a chain of events: configuration loading, data pre-processing, dataset creation, then passing to the Trainer module.
- Module Dependencies:
– Indicate that the “mindspeed_rl” package aggregates domain-specific modules (datasets, models, trainers, workers, utils) and how they interact (e.g., Trainer modules call models, which in turn use loss functions and rollouts).
– Represent that configuration files (from the “configs” folder) feed into the “config_cls” in mindspeed_rl, ensuring proper settings for model training.
- External Integrations:

##### Architectural Patterns & Design Principles
- Modular Design: Emphasize the clear separation of concerns in the mindspeed_rl package (datasets, models, trainers, workers, utils).
- Distributed/Parallel Processing: Highlight the integration of distributed task scheduling via Ray and parallel state management through specialized workers and resharding modules.
- Configurability and Extensibility: The heavy use of YAML/JSON configuration files, along with a config class system, points to a design that is highly configurable and extensible for various algorithms and models.

##### Technologies and Frameworks
- Python as the core language (Python 3.10 is specified).
- Deep learning libraries: torch, torch_npu, apex.
- External frameworks: Ray for distributed training and vllm for efficient inference.
- Ascend-specific tools: NPU drivers, Toolkit, Kernel, NNAL, etc.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interaction"
        CLI["CLI/Task Launcher"]:::cli
        DOC["Documentation"]:::docs
        EX["Usage Examples"]:::docs
    end

    subgraph "Configuration & Data Management"
        CONFIG["Configuration Manager"]:::config
    end

    subgraph "Core RL Engine"
        DSET["Datasets"]:::core
        MODEL["Models"]:::core
        TRAIN["Trainers"]:::core
        WORKER["Workers"]:::core
        UTILS["Utils"]:::core
        SCHED["Distributed Scheduler/Worker Manager"]:::worker
    end

    subgraph "External Dependencies"
        ASCEND["Ascend Ecosystem"]:::external
        ORCHESTRATION["Distributed Orchestration (Ray/vllm)"]:::external
    end

    %% Relationships
    CLI -->|"submit_job"| CONFIG
    CONFIG -->|"load_configs"| DSET
    CONFIG -->|"config_for_training"| TRAIN
    DSET -->|"preprocessed_data"| TRAIN
    MODEL -->|"model_def"| TRAIN
    UTILS -->|"support"| TRAIN
    TRAIN -->|"schedule_workers"| SCHED
    SCHED -->|"launch_workers"| WORKER
    TRAIN -->|"ascend_integration"| ASCEND
    WORKER -->|"npu_support"| ASCEND
    TRAIN -->|"ray_integration"| ORCHESTRATION
    WORKER -->|"parallel_processing"| ORCHESTRATION
    DOC ---|"guide"| CLI
    EX ---|"samples"| CLI
    DOC ---|"config_examples"| CONFIG

    %% Click Events
    click CLI "https://github.com/ascend/mindspeed-rl/tree/master/cli/"
    click CONFIG "https://github.com/ascend/mindspeed-rl/tree/master/configs/"
    click DSET "https://github.com/ascend/mindspeed-rl/tree/master/mindspeed_rl/datasets/"
    click MODEL "https://github.com/ascend/mindspeed-rl/tree/master/mindspeed_rl/models/"
    click TRAIN "https://github.com/ascend/mindspeed-rl/tree/master/mindspeed_rl/trainer/"
    click WORKER "https://github.com/ascend/mindspeed-rl/tree/master/mindspeed_rl/workers/"
    click UTILS "https://github.com/ascend/mindspeed-rl/tree/master/mindspeed_rl/utils/"
    click SCHED "https://github.com/ascend/mindspeed-rl/tree/master/mindspeed_rl/workers/scheduler/"
    click DOC "https://github.com/ascend/mindspeed-rl/tree/master/docs/"
    click EX "https://github.com/ascend/mindspeed-rl/tree/master/examples/"

    %% Styles
    classDef cli fill:#fdd,stroke:#8B0000,stroke-width:2px;
    classDef config fill:#ffcc99,stroke:#D2691E,stroke-width:2px;
    classDef core fill:#ccf,stroke:#00008B,stroke-width:2px;
    classDef worker fill:#cfc,stroke:#006400,stroke-width:2px;
    classDef docs fill:#efe,stroke:#228B22,stroke-width:2px;
    classDef external fill:#fcf,stroke:#8B008B,stroke-width:2px;
```
