# rosieyzh/openrlhf-pretrain architecture

> Architecture diagram of the GitHub repository rosieyzh/openrlhf-pretrain, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rosieyzh/openrlhf-pretrain
- Repository: https://github.com/rosieyzh/openrlhf-pretrain

## Overview

##### Project type and purpose
– This is a research‐oriented machine-learning pipeline rather than a standard web application. It orchestrates three major stages:
- Pretraining a large language model (via the AllenAI OLMo codebase)
- RL-based post-training (via OpenRLHF + Ray/PPO + Expert Iteration)
- Inference and evaluation (pass@k, “math-verify,” benchmarking)
– Its purpose is to run end-to-end experiments (“Echo Chamber”) exploring how RL fine-tuning amplifies behaviors learned during pretraining.

##### Top-level structure
– OLMo/: the core pretraining library (data loaders, model, training loop, checkpoints, Dockerfiles, CI)
– openrlhf/: the RL-fine-tuning library (CLI interfaces, PPO/EI trainers, distributed Ray launcher, DeepSpeed support)
– pretraining/: wrapper scripts and SLURM configs for launching OLMo on HPC clusters
– inference/: scripts for running model inference, collecting pass@k, invoking math-verify, producing JSON results
– configs/: YAML/JSON files driving both pretraining and fine-tuning sweeps
– scripts/: auxiliary Python/Shell helpers for SLURM array launches, dataset conversion, checkpoint manipulation
– examples/, dockerfile/: sample usage, container definition
– pyproject.toml, requirements.txt, submodules: glue to subprojects and dependencies

##### README insights
– Describes a two-phase workflow:
a. Pretraining with OLMo using SLURM scripts in pretraining/
b. RL fine-tuning (PPO, Expert Iteration) via openrlhf/ interfaces and scripts/ppo_sweep.py
– Inference uses inference/run_inference_all.py → math-verify integration, optionally query_gpt.py for qualitative analysis.

– Data storage (raw datasets, tokenized datasets, checkpoint storage)
– Configuration manager (YAML/JSON files, scripts that parse them)
– Pretraining engine (OLMo library + SLURM wrapper + Docker)
– RL fine-tune engine (OpenRLHF library + Ray cluster + SLURM wrapper + Docker)
– Inference & evaluation engine (scripts invoking model + math-verify + logging)
– Orchestration layer (SLURM scheduler, Ray, Docker containers, dispatch scripts)
– External services (WandB for logging, HPC cluster, S3 bucket for checkpoint storage)
1. Configs → SLURM job submission (pretraining scripts)
2. SLURM → containerized OLMo run → raw data → tokenization → model checkpoints
3. Checkpoints (+ configs) → RL fine-tuning scripts → Ray actor cluster → new checkpoints
4. Fine-tuned checkpoints → inference scripts → generation outputs (JSON)
5. Outputs → math-verify → aggregated pass@k metrics → WandB or files
c. Architectural patterns:
– Modular pipeline: clearly separated stages with hand-off via saved checkpoints
– Config-driven workflows: all hyperparameters and sweep definitions live in YAML/JSON
– Distributed compute: SLURM for multi-node, Ray for distributed RL, Docker for reproducibility
– CLI interfaces: each major stage exposes a Python CLIs under openrlhf/cli and inference/
d. Key technologies and roles:
– Python + PyTorch (OLMo, OpenRLHF)
– Ray (distributed RL actors)
– SLURM / Docker (cluster orchestration and containerization)
– vLLM (model serving for inference)
– math-verify (correctness checking during eval)
– WandB or filesystem for metrics logging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Manager
    Configs["Configuration Manager\n(configs/)"]:::config

    %% Pretraining Stage
    subgraph "Pretraining Stage"
        Pretrain_script["SLURM Wrappers & Data Conv\n(pretraining/scripts/, pretraining/data/)"]:::pre
        SLURM["SLURM Scheduler"]:::infra
        Docker["Docker Containers"]:::infra
        Pretrain["Pretraining Engine\nOLMo core lib"]:::pre
        Dockerfile["Dockerfile\n(dockerfile/Dockerfile)"]:::infra
        DataStorage["Raw & Tokenized Data\nStorage"]:::storage
    end

    %% Checkpoint Store
    CheckpointStore["Checkpoint Store\n(S3/Filesystem)"]:::storage

    %% RL Fine-tuning Stage
    subgraph "RL Fine-tuning Stage"
        SweepPPO["PPO Sweep\n(scripts/ppo_sweep.py)"]:::rl
        SweepEI["EI Sweep\n(scripts/ei_sweep.py)"]:::rl
        RL["RL Fine-tuning Engine\nOpenRLHF lib"]:::rl
        RL_CLI["RL CLI Interfaces\n(openrlhf/cli/)"]:::rl
        RL_Trainer["RL Trainers\n(openrlhf/trainer/)"]:::rl
        Ray_Launch["Ray Launcher\n(openrlhf/trainer/ray/launcher.py)"]:::rl
    end

    %% Inference & Evaluation Stage
    subgraph "Inference & Evaluation Stage"
        Inference_Config["Inference Config\n(configs/inference/example_1b.yaml)"]:::config
        RunAll["run_inference_all.py"]:::inf
        QueryGPT["query_gpt.py"]:::inf
        QueryGPTPass["query_gpt_pass.py"]:::inf
        MathVerify["Math-Verify\n(openrlhf/utils/math_verifier.py)"]:::inf
        Outputs["JSON Outputs & Metrics"]:::storage
    end

    %% Auxiliaries & Examples
    AuxScripts["HPC Helpers\n(scripts/)"]:::infra
    DockerEntrypoint["Docker Entrypoint\n(dockerfile/docker-entrypoint.sh)"]:::infra
    Examples["Example Usage\n(examples/scripts/train_ppo_llama_ray_slurm.sh)"]:::doc
    Glue["Project Glue\n(pyproject.toml,\nrequirements.txt)"]:::doc

    %% External Services
    WandB["WandB / Logging"]:::external
    HPC["HPC Cluster"]:::external

    %% Data Flows
    Configs -->|".yaml configs"| Pretrain_script
    Pretrain_script --> SLURM
    SLURM --> Docker
    Docker --> Pretrain
    Pretrain --> DataStorage
    DataStorage --> Pretrain
    Pretrain -->|".ckpt files"| CheckpointStore

    CheckpointStore --> RL
    Configs --> RL
    SweepPPO --> RL
    SweepEI --> RL
    RL_CLI --> RL
    RL_Trainer --> RL
    Ray_Launch --> RL
    RL -->|".ckpt files"| CheckpointStore

    CheckpointStore --> RunAll
    Inference_Config --> RunAll
    RunAll --> QueryGPT
    RunAll --> QueryGPTPass
    RunAll --> MathVerify
    MathVerify --> Outputs
    Outputs --> WandB

    %% Orchestration & Helpers
    HPC --> SLURM
    DockerEntrypoint --> Docker
    AuxScripts --> SLURM
    Examples --> SLURM
    Glue --> Configs

    %% Click Events
    click Configs "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/configs/"
    click Pretrain "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/OLMo/"
    click Pretrain_script "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/pretraining/scripts/"
    click Dockerfile "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/dockerfile/Dockerfile"
    click RL "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/openrlhf/"
    click RL_CLI "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/openrlhf/cli/"
    click RL_Trainer "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/openrlhf/trainer/"
    click Ray_Launch "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/openrlhf/trainer/ray/launcher.py"
    click SweepPPO "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/scripts/ppo_sweep.py"
    click SweepEI "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/scripts/ei_sweep.py"
    click RunAll "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/inference/run_inference_all.py"
    click QueryGPT "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/inference/query_gpt.py"
    click QueryGPTPass "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/inference/query_gpt_pass.py"
    click MathVerify "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/openrlhf/utils/math_verifier.py"
    click Inference_Config "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/configs/inference/example_1b.yaml"
    click AuxScripts "https://github.com/rosieyzh/openrlhf-pretrain/tree/main/scripts/"
    click DockerEntrypoint "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/dockerfile/docker-entrypoint.sh"
    click Examples "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/examples/scripts/train_ppo_llama_ray_slurm.sh"
    click Glue "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/pyproject.toml"
    click Glue "https://github.com/rosieyzh/openrlhf-pretrain/blob/main/requirements.txt"

    %% Styles
    classDef pre fill:#D0E6FF,stroke:#0B5FFF,color:#000
    classDef rl fill:#DFF7D0,stroke:#0A9400,color:#000
    classDef inf fill:#FFE6C0,stroke:#FF9000,color:#000
    classDef infra fill:#F0D0FF,stroke:#9400A0,color:#000
    classDef storage fill:#E0E0E0,stroke:#888888,color:#000
    classDef config fill:#FFF3B0,stroke:#E6B800,color:#000
    classDef external fill:#F0F0F0,stroke:#AAAAAA,color:#000
    classDef doc fill:#F7E6E6,stroke:#D90000,color:#000
```
