How knoveleng/open-rs works
Project Type and Purpose
• This is a research-oriented Python library and CLI tool (“Open RS”) for fine-tuning small LLMs (1.5B params) via reinforcement learning (GRPO) and evaluating them on math reasoning tasks.
• No UI/frontend: it’s a command-line driven full “ML pipeline” consisting of configuration, training, evaluation, logging, and model/dataset orchestration with Hugging Face and Weights & Biases.
File Structure Analysis
• Top-level scripts & configs:
• setup.py / setup.cfg – package install
• eval.sh – shell wrapper for evaluation
• recipes/accelerate_configs/*.yaml – distributed‐training setups
• recipes/grpo.yaml, data_cleaner.yaml – experiment configs
• Source code: src/open_r1/
• grpo.py – RL training entry point
• evaluate.py – evaluation logic (via lighteval)
• generate.py – generation utilities
• sft.py – supervised fine-tuning baseline
• utils/ – callbacks, logging (wandb), model/dataset loading, Hugging Face hub I/O
• Output & logs:
• logs/ – hierarchical experiment results (JSON)
• assets/ – figures for README
• Dependencies & infra: uses accelerate, vLLM, flash-attn, torch, wandb, huggingface_hub.
Key Architectural Components
• A. Configuration Layer
• YAML configs (recipes/) specify experiment parameters (algorithm, resources, dataset paths).
• accelerate launch CLI orchestrates multi-GPU processes.
• B. Training Engine (GRPO)
• grpo.py imports configs, initializes model+tokenizer, data loader, RL loop.
• Under the hood: vLLM model server, FlashAttention kernels.
• Uses callbacks for checkpointing, wandb logging.
• C. Evaluation Engine
• evaluate.py interfaces with lighteval to run LLM on custom tasks.
• eval.sh orchestrates single/multi-GPU runs, writes JSON results under logs/evals.
• D. Data & Models
• Data: Hugging Face Datasets (open-s1, open-rs, open-deepscaler).
• Models: pull/push via Hugging Face Hub (Open-RS1/2/3, OpenRS-GRPO).
• E. Logging & Visualization
• Wandb for real-time metrics; local JSON logs for batch analyses.
• assets/ holds static plots.
Relationships & Data Flow
1. User runs accelerate launch --config_file recipes/accelerate_configs/... src/open_r1/grpo.py --config recipes/grpo.yaml
→ Config parser → Initializes training engine
→ Data loader fetches dataset (Hugging Face hub)
→ Model instantiation (vLLM/PyTorch)
→ RL loop emits metrics → wandb + local JSON logs → model checkpoints (Hugging Face hub)
2. For evaluation: user runs eval.sh or a direct lighteval vllm ... src/open_r1/evaluate.py
→ load trained model → custom lighteval tasks → generation results → JSON under logs/evals
3. Post-processing / overall metrics: logs/overall folders aggregate experiment runs for visualization in assets/.
Architectural Patterns & Principles
• Modular CLI pattern: separate entry points for training, evaluation, generation, SFT.
• Config-Driven Execution: YAML + accelerate for reproducibility and distributed scalability.
• Callback-Based Logging: plug wandb, checkpointing without cluttering core algorithm.
• Clear separation: Data ingestion → Model compute → Logging/Monitoring → Hub I/O.