How microsoft/rstar works
Project Type and Purpose
• This is an open-source, research-oriented Python codebase that implements an ML pipeline for math reasoning with small language models (SLMs) via Monte Carlo Tree Search (MCTS).
• Its core goal is to generate high-quality reasoning trajectories (deep thinking) using a policy model and a reward model, use those trajectories to produce training data (SFT and RM), train the models, and finally evaluate on math benchmarks.
High-Level Architecture
A. Configuration & Data
B. Core Inference Pipeline (main.py, eval.py)
C. Data Extraction & Training Pipeline (extra_*.py, train/)
D. Evaluation & Results
Main Components
A. Configuration & Data
• config/*.yaml – defines MCTS parameters, beam-search, sampling settings.
• eval_data/*.json – raw math problem datasets.
• few_shots/*.json – prompt templates and few-shot examples for policy initialization.
B. Core Inference
• main.py – orchestrates trajectory generation:
• loads config, model_dir (policy), optionally reward_model_dir.
• instantiates LLM engine (rstar_deepthink/llms/llm_engine.py).
• invokes MCTS or beam search agents (rstar_deepthink/agents/).
• interacts with node abstractions (rstar_deepthink/nodes/).
• writes raw trajectory files.
• run_example.sh – convenience script demonstrating multi-trajectory runs.
• eval.py, eval_output.py – wrappers for Greedy decode and result file evaluation.
C. Data Extraction & Training
• extra_sft_file.py, extra_rm_file.py – parse MCTS trajectory files into SFT and RM datasets.
• train/
• sample_sft_data.py, sample_rm_data.py – subsample data for training.
• train_SFT.py (distributed fine-tuning of the policy model).
• train_RM.py (reward model training via pairwise preference).
• save_rm.py – convert checkpoints to RM format.
D. Utilities & Supporting Code
• utils/ – math answer checking, parsing, generic utilities.
• rstar_deepthink/solver.py – high-level wrapper combining prompts, agents, and node logic.
• rstar_deepthink/tools/python_tool.py – code execution tool for solving subproblems.
Relationships and Data Flow
1. Preprocessing
• Input: eval_data and few_shots JSON.
• Output: few-shot prompts ready for the inference pipeline.
2. Inference Pipeline (main.py)
• Config Loader → LLM Engine → Agent (MCTS/Beam Search) → Node Expansion → Reward Model → Trajectory Storage
3. Extraction Pipeline
• Take raw trajectories → extra_sft_file / extra_rm_file → generate .jsonl of SFT and RM examples
• sample_* scripts → create training subsets
4. Model Training
• Distributed SFT and RM training scripts consume sampled data → produce updated policy and reward model checkpoints
5. Iteration
• New policy and reward models feed back into the inference pipeline for further rounds.
6. Evaluation
• eval.py and eval_output.py apply models to test sets → compute accuracy metrics.
Architectural Patterns & Design Principles
• Modular CLI: each script has a single responsibility (generate, extract, sample, train, evaluate).
• Config-driven execution: all parameters (e.g., GPU allocation, MCTS depth, beam width) are externalized in YAML.
• Clear separation of concerns:
• rstar_deepthink/ for core algorithmic logic
• train/ for training loops
• utils/ for cross-cutting utilities
• Data lineage tracking: raw trajectories → extracted examples → model checkpoints → new trajectories