# ypwang61/one-shot-rlvr architecture

> Architecture diagram of the GitHub repository ypwang61/one-shot-rlvr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ypwang61/one-shot-rlvr
- Repository: https://github.com/ypwang61/one-shot-rlvr

## Overview

##### Project type and purpose
- This is a full‐stack machine‐learning pipeline for “One‐Shot” reinforcement learning on large language models (LLMs).
- It contains two main flows:
a. A training pipeline (built on top of the Verl and RLLM frameworks) to fine‐tune LLMs (e.g., Qwen2.5‐Math‐1.5B) via PPO/SFT with one example.
b. An evaluation pipeline (Qwen2.5‐Eval) to benchmark math reasoning performance using a suite of datasets and a LaTeX‐to‐Sympy converter.

##### File structure
- Top‐level directories:
- verl/ → core training library (models, trainers, controllers, workers, utils)
- Qwen2.5‐Eval/ → evaluation suite (data loaders, graders, latex2sympy, scripts)
- data/ → selection scripts & DSR‐sub training data
- scripts/ → orchestration (bash entrypoints for training & eval)
- examples/ → runnable examples for data preprocessing, generation, PPO/SFT training, Ray setups
- docker/ → container definitions for vLLM and VEMLP
- docs/ → Sphinx documentation, tutorials, advanced extensions
- tests/ → unit, E2E, sanity, GPU utilities, Ray workflows
- Key config files: pyproject.toml, setup.py, requirements*.txt, verl/trainer/config/*.yaml
- CI/CD: .github/workflows for dataset, model, end‐to‐end, Ray tests, formatting, etc.

##### README for additional insights
- Training environment uses Conda + PyTorch + Ray + vLLM + Flash‐attention + WandB + HuggingFace Hub.
- Evaluation environment uses a separate Conda env, Qwen2.5‐Eval/evaluation, latex2sympy, Transformers, vLLM.
- Data selection is driven by historical variance on DSR‐sub via data/data_selection.sh.
- Entrypoints: scripts/train/*.sh → training; Qwen2.5‐Eval/evaluation/sh/*.sh → evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% User Interface & Scripts
  subgraph "User Interface & Scripts"
    style UI fill:#f0f0f0,stroke:#ccc
    script1["training_1.5b_dsr_sub.sh"]:::scripts
    script2["training_1.5b_pi1_r128.sh"]:::scripts
  end

  %% Data Layer
  subgraph "Data Layer"
    style DL fill:#e0f7fa,stroke:#006064
    DSR["One-Shot RLVR Data"]:::data
    selScript["data_selection.py"]:::scripts
    evalData["Evaluation Datasets"]:::data
  end

  %% Training Pipeline
  subgraph "Training Pipeline"
    style TP fill:#fff3e0,stroke:#e65100
    config["Trainer Configs (YAML)"]:::config
    subgraph "Entrypoints"
      style EP fill:#fff8e1,stroke:#ffa000
      ppo["main_ppo.py"]:::compute
      sft["fsdp_sft_trainer.py"]:::compute
      gen["main_generation.py"]:::compute
      evalEP["main_eval.py"]:::compute
      evalAll["main_evaluation_all_checkpoints.py"]:::compute
    end
    subgraph "Trainer Core"
      style TC fill:#ffecb3,stroke:#ff6f00
      coreAlgos["core_algos.py"]:::compute
      rayTrainer["ray_trainer.py"]:::compute
    end
    subgraph "Controllers"
      style Ctrl fill:#fff3e0,stroke:#fb8c00
      baseCtrl["base"]:::compute
      megaCtrl["megatron"]:::compute
      rayBase["ray/base.py"]:::compute
      rayMega["ray/megatron.py"]:::compute
    end
    subgraph "Workers"
      style W fill:#ffe0b2,stroke:#ef6c00
      actor["actor"]:::compute
      critic["critic"]:::compute
      rollout["rollout"]:::compute
      reward["reward_model"]:::compute
      shardMgr["sharding_manager"]:::compute
    end
    subgraph "Models"
      style M fill:#fff3e0,stroke:#ff8f00
      llamaMega["llama/megatron"]:::compute
      qwenMega["qwen2/megatron"]:::compute
      transAdapters["transformers adapters"]:::compute
    end
  end

  %% Utils & Checkpoint & Logging
  subgraph "Utils"
    style U fill:#e8f5e9,stroke:#2e7d32
    ckptMgr["checkpoint_manager.py"]:::compute
    fsdpMgr["fsdp_checkpoint_manager.py"]:::compute
    datasetRL["rl_dataset.py"]:::compute
    datasetRM["rm_dataset.py"]:::compute
    datasetSFT["sft_dataset.py"]:::compute
    loggingUtils["logging_utils.py"]:::compute
    aggLogger["aggregate_logger.py"]:::compute
    distUtils["distributed.py"]:::compute
    rayUtils["ray_utils.py"]:::compute
  end

  %% Evaluation Pipeline
  subgraph "Evaluation Pipeline"
    style EP2 fill:#e8eaf6,stroke:#3949ab
    evalScript["evaluate.py"]:::eval
    subgraph "Eval Components"
      style EC fill:#e3f2fd,stroke:#1e88e5
      evalDataLoader["data_loader.py"]:::eval
      grader["grader.py"]:::eval
      rmEval["rm_maj_eval.py"]:::eval
      trajectory["trajectory.py"]:::eval
      latex2sym["latex2sympy"]:::eval
      mathEval["math_eval.py"]:::eval
      modelUtils["model_utils.py"]:::eval
      mathUtils["math_utils.py"]:::eval
      pyExec["python_executor.py"]:::eval
    end
  end

  %% Infrastructure & Services
  subgraph "Infrastructure & Services"
    style IS fill:#f3e5f5,stroke:#6a1b9a
    Ray["Ray Cluster"]:::external
    PyTorch["PyTorch & FSDP"]:::external
    vLLM["vLLM"]:::external
    Hub["HuggingFace Hub"]:::external
    WandB["Weights & Biases"]:::external
    Docker["Docker Containers"]:::external
  end

  %% Connections
  selScript -->|selects data| DSR
  selScript -->|feeds loader| datasetRL
  DSR -->|shard created| datasetRL
  script1 -->|invokes| ppo
  script2 -->|invokes| sft
  ppo --> config
  sft --> config
  ppo --> coreAlgos
  sft --> coreAlgos
  ppo --> rayTrainer
  rayTrainer --> baseCtrl
  baseCtrl --> actor
  baseCtrl --> critic
  baseCtrl --> rollout
  baseCtrl --> reward
  baseCtrl --> shardMgr
  actor -->|inference| vLLM
  rollout -->|trajectory| coreAlgos
  reward --> coreAlgos
  coreAlgos --> ckptMgr
  ckptMgr --> Hub
  coreAlgos --> loggingUtils
  loggingUtils --> WandB

  evalScript -->|loads checkpoints| ckptMgr
  evalScript --> evalDataLoader
  evalDataLoader --> latex2sym
  latex2sym --> grader
  grader -->|metrics| WandB
  pyExec -->|executes| evalScript

  %% Click Events
  click DSR "https://github.com/ypwang61/one-shot-rlvr/tree/main/data/train/one_shot_rlvr"
  click selScript "https://github.com/ypwang61/one-shot-rlvr/blob/main/data/data_selection.py"
  click evalData "https://github.com/ypwang61/one-shot-rlvr/tree/main/Qwen2.5‐Eval/evaluation/data"
  click script1 "https://github.com/ypwang61/one-shot-rlvr/blob/main/scripts/train/training_1.5b_dsr_sub.sh"
  click script2 "https://github.com/ypwang61/one-shot-rlvr/blob/main/scripts/train/training_1.5b_pi1_r128.sh"
  click ppo "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/main_ppo.py"
  click sft "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/fsdp_sft_trainer.py"
  click gen "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/main_generation.py"
  click evalEP "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/main_eval.py"
  click evalAll "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/main_evaluation_all_checkpoints.py"
  click config "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/config/ppo_trainer.yaml"
  click config "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/config/ppo_megatron_trainer.yaml"
  click config "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/config/sft_trainer.yaml"
  click config "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/config/generation.yaml"
  click config "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/config/evaluation.yaml"
  click coreAlgos "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/ppo/core_algos.py"
  click rayTrainer "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/trainer/ppo/ray_trainer.py"
  click baseCtrl "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/single_controller/base"
  click megaCtrl "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/single_controller/base/megatron"
  click rayBase "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/single_controller/ray/base.py"
  click rayMega "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/single_controller/ray/megatron.py"
  click actor "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/workers/actor"
  click critic "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/workers/critic"
  click rollout "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/workers/rollout"
  click reward "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/workers/reward_model"
  click shardMgr "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/workers/sharding_manager"
  click llamaMega "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/models/llama/megatron"
  click qwenMega "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/models/qwen2/megatron"
  click transAdapters "https://github.com/ypwang61/one-shot-rlvr/tree/main/verl/models/transformers"
  click ckptMgr "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/checkpoint/checkpoint_manager.py"
  click fsdpMgr "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/checkpoint/fsdp_checkpoint_manager.py"
  click datasetRL "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/dataset/rl_dataset.py"
  click datasetRM "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/dataset/rm_dataset.py"
  click datasetSFT "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/dataset/sft_dataset.py"
  click loggingUtils "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/logging_utils.py"
  click aggLogger "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/logger/aggregate_logger.py"
  click distUtils "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/distributed.py"
  click rayUtils "https://github.com/ypwang61/one-shot-rlvr/blob/main/verl/utils/ray_utils.py"
  click evalScript "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/evaluate.py"
  click evalDataLoader "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/data_loader.py"
  click grader "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/grader.py"
  click rmEval "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/rm_maj_eval.py"
  click trajectory "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/trajectory.py"
  click latex2sym "https://github.com/ypwang61/one-shot-rlvr/tree/main/Qwen2.5‐Eval/evaluation/latex2sympy"
  click mathEval "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/math_eval.py"
  click modelUtils "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/model_utils.py"
  click mathUtils "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/math_utils.py"
  click pyExec "https://github.com/ypwang61/one-shot-rlvr/blob/main/Qwen2.5‐Eval/evaluation/python_executor.py"

  %% Styles
  classDef data fill:#e0f7fa,stroke:#006064
  classDef scripts fill:#fce4ec,stroke:#c2185b
  classDef compute fill:#fff3e0,stroke:#ef6c00
  classDef config fill:#e8eaf6,stroke:#3949ab
  classDef eval fill:#e3f2fd,stroke:#1e88e5
  classDef external fill:#f3e5f5,stroke:#6a1b9a
```
