# prime-rl/prime architecture

> Architecture diagram of the GitHub repository prime-rl/prime, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prime-rl/prime
- Repository: https://github.com/prime-rl/prime

## Overview

##### Project type and purpose
- This is a research-grade reinforcement‐learning pipeline built around “Process Reinforcement through Implicit Rewards” (PRIME).
- It comprises three major subsystems:
- data_preprocessing – preparing and filtering math/coding datasets
- training – an online RL loop that trains both a policy LLM and an implicit Process Reward Model (PRM) using PPO and RLOO
- eval – post‐training benchmark scripts against multiple datasets (HumanEval, LeetCode, AMC/AIME, etc.)

##### Key modules (top‐level directories)
- data_preprocessing/
- math_util – normalization, grading utilities
- stage1_filter → stage4_judge – sequential scripts to filter, format, merge, and judge examples
- training/
- verl/ – extension of the veRL framework with model loaders, controllers (Ray, Megatron, vLLM), distributed settings, trainer entrypoints (main_sft, main_ppo, main_eval)
- scripts & examples – data_prepare, run_prime_main.sh
- requirements & setup
- eval/
- Coding/ – subfolders for human_eval, LeetCode, LiveCodeBench runner
- Math/ – Qwen25‐Math evaluation with LaTeX→Sympy parser & grader, plus AMC, AIME, etc.
- utils/ – shared evaluation, execution, grading, data loading

##### Detail
- Configuration files ➔ controller initializations
- Versioned model repos on Hugging Face
- Scripts (run.sh, main_ppo.py, evaluate_leetcode.py) at the edges as entrypoints

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preprocessing
    subgraph "Data Preprocessing" 
        direction TB
        DataPreprocessing["Overall Data Preprocessing"]:::blue
        MathUtil["Math Util"]:::blue
        Stage1_Filter["Stage1 Filter"]:::blue
        Stage2_Format["Stage2 Format"]:::blue
        Stage3_Merge["Stage3 Merge"]:::blue
        Stage4_Judge["Stage4 Judge"]:::blue
    end

    %% Training Pipeline
    subgraph "Training Pipeline" 
        direction TB
        verlExt["veRL Extension"]:::green
        subgraph "Models & Inference"
            direction TB
            PolicyModel["Policy Model (πθ)"]:::green
            PRMWorkers["PRM Workers"]:::green
            RewardScore["PRM Scoring Logic"]:::green
            InferenceEngine["Inference Engine (vLLM)"]:::green
        end
        subgraph "Controllers"
            direction TB
            RayController["Ray Controller"]:::green
            MegatronController["Megatron Controller"]:::green
        end
        subgraph "Training Loop"
            direction TB
            Rollouts["Generate Rollouts"]:::green
            PRMScorer["PRM Scorer"]:::green
            OutcomeVerifier["Outcome Verifier"]:::green
            RewardCombiner["Reward Combiner & Advantage Estimator"]:::green
            PPOOptimizer["PPO Optimizer"]:::green
        end
        subgraph "Entrypoints & Config"
            direction TB
            main_sft["main_sft (SFT)"]:::green
            main_ppo["main_ppo"]:::green
            main_generation["main_generation"]:::green
            main_eval["main_eval"]:::green
            PPOAlgos["PPO Core Algos"]:::green
            ConfigDir["Trainer Configs"]:::green
            SharedUtils["Shared Training Utils"]:::green
        end
    end

    %% Evaluation Suite
    subgraph "Evaluation Suite" 
        direction TB
        EvalCoding["Coding Tasks"]:::orange
        EvalMath["Math Tasks"]:::orange
        EvalUtils["Shared Eval Utils"]:::orange
    end

    %% External Services
    subgraph "External Services" 
        direction TB
        HFHub["Hugging Face Model Hub"]:::grey
        RayCluster["Ray Cluster"]:::grey
        GPUNodes["GPU Nodes"]:::grey
    end

    %% Data Flow Arrows
    DataPreprocessing -->|filtered prompts| verlExt
    MathUtil --> DataPreprocessing
    Stage1_Filter --> Stage2_Format --> Stage3_Merge --> Stage4_Judge --> DataPreprocessing

    verlExt --> PolicyModel
    PolicyModel --> InferenceEngine
    InferenceEngine --> Rollouts
    Rollouts --> PRMScorer
    Rollouts --> OutcomeVerifier
    PRMScorer --> RewardCombiner
    OutcomeVerifier --> RewardCombiner
    RewardCombiner --> PPOOptimizer
    PPOOptimizer --> PolicyModel

    RayController --> PPOOptimizer
    MegatronController --> PPOOptimizer
    ConfigDir --> RayController
    ConfigDir --> MegatronController
    SharedUtils --> verlExt

    PPOOptimizer --> main_ppo
    main_sft --> verlExt
    main_generation --> EvalCoding
    main_eval --> EvalMath

    PolicyModel -->|final model| EvalCoding
    PolicyModel -->|final model| EvalMath
    EvalCoding --> EvalUtils
    EvalMath --> EvalUtils

    HFHub --> PolicyModel
    RayCluster --> RayController
    GPUNodes --> MegatronController

    %% Click Events
    click DataPreprocessing "https://github.com/prime-rl/prime/tree/main/data_preprocessing/"
    click MathUtil "https://github.com/prime-rl/prime/tree/main/data_preprocessing/math_util/"
    click Stage1_Filter "https://github.com/prime-rl/prime/blob/main/data_preprocessing/stage1_filter.py"
    click Stage2_Format "https://github.com/prime-rl/prime/blob/main/data_preprocessing/stage2_format_choice.py"
    click Stage3_Merge "https://github.com/prime-rl/prime/blob/main/data_preprocessing/stage3_merge.py"
    click Stage4_Judge "https://github.com/prime-rl/prime/blob/main/data_preprocessing/stage4_judge.py"
    click verlExt "https://github.com/prime-rl/prime/tree/main/training/verl/"
    click PolicyModel "https://github.com/prime-rl/prime/tree/main/training/verl/models/llama/"
    click PRMWorkers "https://github.com/prime-rl/prime/tree/main/training/verl/workers/reward_model/"
    click RewardScore "https://github.com/prime-rl/prime/blob/main/training/verl/utils/reward_score/prime.py"
    click InferenceEngine "https://github.com/prime-rl/prime/tree/main/training/verl/third_party/vllm/"
    click RayController "https://github.com/prime-rl/prime/tree/main/training/verl/single_controller/ray/"
    click MegatronController "https://github.com/prime-rl/prime/tree/main/training/verl/single_controller/base/megatron/"
    click main_sft "https://github.com/prime-rl/prime/blob/main/training/verl/trainer/fsdp_sft_trainer.py"
    click main_ppo "https://github.com/prime-rl/prime/blob/main/training/verl/trainer/main_ppo.py"
    click main_generation "https://github.com/prime-rl/prime/blob/main/training/verl/trainer/main_generation.py"
    click main_eval "https://github.com/prime-rl/prime/blob/main/training/verl/trainer/main_eval.py"
    click PPOAlgos "https://github.com/prime-rl/prime/blob/main/training/verl/trainer/ppo/core_algos.py"
    click ConfigDir "https://github.com/prime-rl/prime/tree/main/training/verl/trainer/config/"
    click SharedUtils "https://github.com/prime-rl/prime/blob/main/training/verl/utils/config.py"
    click EvalCoding "https://github.com/prime-rl/prime/tree/main/eval/Coding/"
    click EvalMath "https://github.com/prime-rl/prime/tree/main/eval/Math/"
    click EvalUtils "https://github.com/prime-rl/prime/tree/main/eval/utils/"

    %% Styles
    classDef blue fill:#D0E6FF,stroke:#003366,color:#003366
    classDef green fill:#D0FFD8,stroke:#006600,color:#006600
    classDef orange fill:#FFE6C2,stroke:#CC6600,color:#CC6600
    classDef grey fill:#F2F2F2,stroke:#666666,color:#666666
```
