# agentica-project/rllm architecture

> Architecture diagram of the GitHub repository agentica-project/rllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/agentica-project/rllm
- Repository: https://github.com/agentica-project/rllm

## Overview

Based on our analysis of the file tree and README, this project is a reinforcement learning (RL) framework for fine-tuning large language models (LLMs) applied to both coding and mathematical tasks. It provides end‐to‐end support from data preprocessing (for code and math datasets) and reward functions (for both domains), to multi-node RL training and evaluation. In other words, it’s an open-source toolkit for democratizing RL for LLMs, integrating a variety of modules from data handling to distributed training engines (using, for example, Ray) and specialized pipelines (like the verl-pipe extension).

- Data Layer:
- Raw Data & Preprocessing: Represent the directories under rllm/data (with subfolders train, test, and preprocess for code and math). Note that there are individual preprocessing notebooks and scripts (e.g., in rllm/data/preprocess/code and rllm/data/preprocess/math) that convert raw JSON to Parquet files.
- Reward Functions:
- Rewards module (rllm/rewards) which includes separate reward functions for code (e.g., code_reward) and math (math_reward) as well as supporting utilities.
- Core Library & Utilities:
- Main package (rllm) with global settings (globals.py), utility functions (utils.py), and system prompts. This forms the central “engine” used by training, evaluation, and tools.
- Tools & Extensions:
- Training & Evaluation Pipelines:
- Training scripts (inside scripts/deepcoder and scripts/deepscaler) and the evaluation scripts (scripts/eval). These scripts orchestrate both single-node and multi-node (Ray-based) training and evaluation.
- Distributed & Parallel Training/System Integration:
- Integration with the verl modules (and its pipeline enhancement “verl-pipe”) which seem to extend the core RL training on top of an adapted version of an RLHF library.
- Multi-node training setup that uses Ray (with instructions for head node and worker nodes). This represents how the training tasks are scaled.
- Testing & Examples:
- Don’t forget to include a component representing the tests (in tests/ and in verl/tests) that validate the reward computations, data pipelines, and model generation.
- External Dependencies & Integrations:
- References to external systems such as Hugging Face (for model hosting and datasets), Wandb (for training logs), and cloud storage (Google Drive for evaluation logs).

##### Relationships & Interactions
- Data Flow:
- How raw datasets (rllm/data/train and rllm/data/test) are processed through the preprocessing notebooks/scripts.
- That processed data feeds into the training pipeline.
- Reward Signal Path:
- How the preprocessed data and model outputs are evaluated by reward functions from rllm/rewards. The resulting reward signals are then looped back to influence the reinforcement learning training.
- Training & Evaluation Flow:
- Control flow from training scripts (in scripts/*/train) which interact with the core library, distributed training (Ray and verl), and finally the evaluation module that uses evaluation scripts.
- Distributed Coordination:
- Pipeline Acceleration:

##### Architectural Patterns & Design Principles
- Modular Separation:
- Distributed & Parallel Processing:
- Components such as Ray clusters, asynchronous pipelines, and multi-tool integrations that are central to scaling the RL training.
- Reusability & Extensibility:
- “tools” layer that provides specialized functions and integrations (e.g., code evaluation, math calculation, web searches) that can be extended or swapped out.
- Clear Interface Boundaries:
- Interfaces between data preprocessing, reward functions, training scripts, and evaluation. This delineates responsibilities and dependencies in the architecture.

##### Detail the Data & Control Flow
- Demonstrate the preprocessing step leading to standardized Parquet format.
- How the training scripts call the core rllm functions (incorporating reward functions and system prompts).
- Distributed communication layer (Ray cluster and verl modules) orchestrating multi-node training.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DataLayer["Raw Data (rllm/data)"]:::datalayer
        DataTrain["Data Train (rllm/data/train)"]:::datalayer
        DataTest["Data Test (rllm/data/test)"]:::datalayer
        Preprocess["Preprocessing (rllm/data/preprocess)"]:::datalayer
        CodePreprocess["Code Preprocessing (rllm/data/preprocess/code)"]:::datalayer
        MathPreprocess["Math Preprocessing (rllm/data/preprocess/math)"]:::datalayer
    end

    %% Reward Functions
    subgraph "Reward Functions"
        RewardModule["Reward Module (rllm/rewards)"]:::rewards
    end

    %% Core Library & Utilities
    subgraph "Core Library & Utilities"
        CoreEngine["Core Engine (rllm)"]:::core
        Globals["Globals (rllm/globals.py)"]:::core
        Utils["Utils (rllm/utils.py)"]:::core
        SystemPrompts["System Prompts (rllm/system_prompts.py)"]:::core
    end

    %% Tools & Extensions
    subgraph "Tools & Extensions"
        ToolsMain["Tools (rllm/tools)"]:::tools
        CodeTools["Code Tools (rllm/tools/code_tools)"]:::tools
        MathTools["Math Tools (rllm/tools/math_tools)"]:::tools
        WebTools["Web Tools (rllm/tools/web_tools)"]:::tools
    end

    %% Training & Evaluation Pipelines
    subgraph "Training & Evaluation Pipelines"
        Deepcoder["Deepcoder Training (scripts/deepcoder)"]:::training
        Deepscaler["Deepscaler Training (scripts/deepscaler)"]:::training
        Eval["Evaluation (scripts/eval)"]:::training
        DataPrep["Data Preparation (scripts/data)"]:::training
    end

    %% Distributed & Parallel Training/System Integration
    subgraph "Distributed & Parallel Training/System Integration"
        DistributedEngine["Distributed Engine (Verl)"]:::distributed
        VerlPipe["Verl-pipe (scripts/pipeline)"]:::distributed
        RayHead["Ray Head (verl/verl/single_controller)"]:::distributed
        RayWorker["Ray Worker"]:::distributed
    end

    %% Testing & Examples
    subgraph "Testing & Examples"
        TestsMain["Tests (tests)"]:::tests
        DistributedTests["Distributed Tests (verl/tests)"]:::tests
    end

    %% External Dependencies
    subgraph "External Dependencies"
        HF["HuggingFace"]:::external
        Wandb["Wandb"]:::external
        GDrive["Google Drive"]:::external
    end

    %% Connections within Data Layer
    DataLayer --> DataTrain
    DataLayer --> DataTest
    DataLayer --> Preprocess
    Preprocess --> CodePreprocess
    Preprocess --> MathPreprocess

    %% Data flow from Preprocessing to Training Pipeline
    CodePreprocess -->|"processed"| DataPrep
    MathPreprocess -->|"processed"| DataPrep
    DataPrep --> Deepcoder
    DataPrep --> Deepscaler

    %% Training script interactions with Core and Reward modules
    Deepcoder -->|"uses"| CoreEngine
    Deepscaler -->|"uses"| CoreEngine
    Deepcoder -->|"evaluated_by"| RewardModule
    Deepscaler -->|"evaluated_by"| RewardModule
    RewardModule -->|"feedback"| Deepcoder
    RewardModule -->|"feedback"| Deepscaler

    %% Training to Evaluation flow
    Deepcoder --> Eval
    Deepscaler --> Eval

    %% Distributed & Parallel Training integration
    Deepcoder -->|"distributed_call"| DistributedEngine
    Deepscaler -->|"distributed_call"| DistributedEngine
    DistributedEngine --> VerlPipe
    DistributedEngine --> RayHead
    RayHead --> RayWorker
    RayWorker --> RayHead

    %% Core and Tools connections
    CoreEngine --> ToolsMain
    ToolsMain --> Deepcoder
    ToolsMain --> Deepscaler

    %% External Dependencies
    HF --> Deepcoder
    HF --> Deepscaler
    Wandb --> Deepcoder
    Wandb --> Deepscaler
    GDrive --> Eval

    %% Testing connections
    RewardModule --> TestsMain
    CoreEngine --> TestsMain
    DistributedEngine --> DistributedTests

    %% Click Events for Data Layer
    click DataLayer "https://github.com/agentica-project/rllm/tree/main/rllm/data"
    click DataTrain "https://github.com/agentica-project/rllm/tree/main/rllm/data/train"
    click DataTest "https://github.com/agentica-project/rllm/tree/main/rllm/data/test"
    click Preprocess "https://github.com/agentica-project/rllm/tree/main/rllm/data/preprocess"
    click CodePreprocess "https://github.com/agentica-project/rllm/tree/main/rllm/data/preprocess/code"
    click MathPreprocess "https://github.com/agentica-project/rllm/tree/main/rllm/data/preprocess/math"

    %% Click Events for Reward Functions
    click RewardModule "https://github.com/agentica-project/rllm/tree/main/rllm/rewards"

    %% Click Events for Core Library & Utilities
    click Globals "https://github.com/agentica-project/rllm/blob/main/rllm/globals.py"
    click Utils "https://github.com/agentica-project/rllm/blob/main/rllm/utils.py"
    click SystemPrompts "https://github.com/agentica-project/rllm/blob/main/rllm/system_prompts.py"

    %% Click Events for Tools & Extensions
    click ToolsMain "https://github.com/agentica-project/rllm/tree/main/rllm/tools"
    click CodeTools "https://github.com/agentica-project/rllm/tree/main/rllm/tools/code_tools"
    click MathTools "https://github.com/agentica-project/rllm/tree/main/rllm/tools/math_tools"
    click WebTools "https://github.com/agentica-project/rllm/tree/main/rllm/tools/web_tools"

    %% Click Events for Training & Evaluation Pipelines
    click Deepcoder "https://github.com/agentica-project/rllm/tree/main/scripts/deepcoder"
    click Deepscaler "https://github.com/agentica-project/rllm/tree/main/scripts/deepscaler"
    click Eval "https://github.com/agentica-project/rllm/tree/main/scripts/eval"
    click DataPrep "https://github.com/agentica-project/rllm/tree/main/scripts/data"

    %% Click Events for Distributed & Parallel Training/System Integration
    click DistributedEngine "https://github.com/agentica-project/rllm/tree/main/verl/"
    click VerlPipe "https://github.com/agentica-project/rllm/tree/main/scripts/pipeline"
    click RayHead "https://github.com/agentica-project/rllm/tree/main/verl/verl/single_controller"
    click RayWorker "https://github.com/agentica-project/rllm/tree/main/verl/tests/ray"

    %% Click Events for Testing & Examples
    click TestsMain "https://github.com/agentica-project/rllm/tree/main/tests"
    click DistributedTests "https://github.com/agentica-project/rllm/tree/main/verl/tests"

    %% Styling definitions
    classDef datalayer fill:#FFEECC,stroke:#333,stroke-width:2px;
    classDef rewards fill:#FFD6CC,stroke:#333,stroke-width:2px;
    classDef core fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef tools fill:#E6CCFF,stroke:#333,stroke-width:2px;
    classDef training fill:#CCE5FF,stroke:#333,stroke-width:2px;
    classDef distributed fill:#E0E0E0,stroke:#333,stroke-width:2px;
    classDef tests fill:#FFFFCC,stroke:#333,stroke-width:2px;
    classDef external fill:#DFF0D8,stroke:#333,stroke-width:2px;
```
