How rosieyzh/openrlhf-pretrain works
Project type and purpose
– This is a research‐oriented machine-learning pipeline rather than a standard web application. It orchestrates three major stages:
• Pretraining a large language model (via the AllenAI OLMo codebase)
• RL-based post-training (via OpenRLHF + Ray/PPO + Expert Iteration)
• Inference and evaluation (pass@k, “math-verify,” benchmarking)
– Its purpose is to run end-to-end experiments (“Echo Chamber”) exploring how RL fine-tuning amplifies behaviors learned during pretraining.
Top-level structure
– OLMo/: the core pretraining library (data loaders, model, training loop, checkpoints, Dockerfiles, CI)
– openrlhf/: the RL-fine-tuning library (CLI interfaces, PPO/EI trainers, distributed Ray launcher, DeepSpeed support)
– pretraining/: wrapper scripts and SLURM configs for launching OLMo on HPC clusters
– inference/: scripts for running model inference, collecting pass@k, invoking math-verify, producing JSON results
– configs/: YAML/JSON files driving both pretraining and fine-tuning sweeps
– scripts/: auxiliary Python/Shell helpers for SLURM array launches, dataset conversion, checkpoint manipulation
– examples/, dockerfile/: sample usage, container definition
– pyproject.toml, requirements.txt, submodules: glue to subprojects and dependencies
README insights
– Describes a two-phase workflow:
a. Pretraining with OLMo using SLURM scripts in pretraining/
b. RL fine-tuning (PPO, Expert Iteration) via openrlhf/ interfaces and scripts/ppo_sweep.py
– Inference uses inference/run_inference_all.py → math-verify integration, optionally query_gpt.py for qualitative analysis.
– Data storage (raw datasets, tokenized datasets, checkpoint storage)
– Configuration manager (YAML/JSON files, scripts that parse them)
– Pretraining engine (OLMo library + SLURM wrapper + Docker)
– RL fine-tune engine (OpenRLHF library + Ray cluster + SLURM wrapper + Docker)
– Inference & evaluation engine (scripts invoking model + math-verify + logging)
– Orchestration layer (SLURM scheduler, Ray, Docker containers, dispatch scripts)
– External services (WandB for logging, HPC cluster, S3 bucket for checkpoint storage)
1. Configs → SLURM job submission (pretraining scripts)
2. SLURM → containerized OLMo run → raw data → tokenization → model checkpoints
3. Checkpoints (+ configs) → RL fine-tuning scripts → Ray actor cluster → new checkpoints
4. Fine-tuned checkpoints → inference scripts → generation outputs (JSON)
5. Outputs → math-verify → aggregated pass@k metrics → WandB or files
c. Architectural patterns:
– Modular pipeline: clearly separated stages with hand-off via saved checkpoints
– Config-driven workflows: all hyperparameters and sweep definitions live in YAML/JSON
– Distributed compute: SLURM for multi-node, Ray for distributed RL, Docker for reproducibility
– CLI interfaces: each major stage exposes a Python CLIs under openrlhf/cli and inference/
d. Key technologies and roles:
– Python + PyTorch (OLMo, OpenRLHF)
– Ray (distributed RL actors)
– SLURM / Docker (cluster orchestration and containerization)
– vLLM (model serving for inference)
– math-verify (correctness checking during eval)
– WandB or filesystem for metrics logging