How ypwang61/one-shot-rlvr works
Project type and purpose
• This is a full‐stack machine‐learning pipeline for “One‐Shot” reinforcement learning on large language models (LLMs).
• It contains two main flows:
a. A training pipeline (built on top of the Verl and RLLM frameworks) to fine‐tune LLMs (e.g., Qwen2.5‐Math‐1.5B) via PPO/SFT with one example.
b. An evaluation pipeline (Qwen2.5‐Eval) to benchmark math reasoning performance using a suite of datasets and a LaTeX‐to‐Sympy converter.
File structure
• Top‐level directories:
• verl/ → core training library (models, trainers, controllers, workers, utils)
• Qwen2.5‐Eval/ → evaluation suite (data loaders, graders, latex2sympy, scripts)
• data/ → selection scripts & DSR‐sub training data
• scripts/ → orchestration (bash entrypoints for training & eval)
• examples/ → runnable examples for data preprocessing, generation, PPO/SFT training, Ray setups
• docker/ → container definitions for vLLM and VEMLP
• docs/ → Sphinx documentation, tutorials, advanced extensions
• tests/ → unit, E2E, sanity, GPU utilities, Ray workflows
• Key config files: pyproject.toml, setup.py, requirements*.txt, verl/trainer/config/*.yaml
• CI/CD: .github/workflows for dataset, model, end‐to‐end, Ray tests, formatting, etc.
README for additional insights
• Training environment uses Conda + PyTorch + Ray + vLLM + Flash‐attention + WandB + HuggingFace Hub.
• Evaluation environment uses a separate Conda env, Qwen2.5‐Eval/evaluation, latex2sympy, Transformers, vLLM.
• Data selection is driven by historical variance on DSR‐sub via data/data_selection.sh.
• Entrypoints: scripts/train/*.sh → training; Qwen2.5‐Eval/evaluation/sh/*.sh → evaluation.