How prime-rl/prime works
Project type and purpose
• This is a research-grade reinforcement‐learning pipeline built around “Process Reinforcement through Implicit Rewards” (PRIME).
• It comprises three major subsystems:
• data_preprocessing – preparing and filtering math/coding datasets
• training – an online RL loop that trains both a policy LLM and an implicit Process Reward Model (PRM) using PPO and RLOO
• eval – post‐training benchmark scripts against multiple datasets (HumanEval, LeetCode, AMC/AIME, etc.)
Key modules (top‐level directories)
• data_preprocessing/
• math_util – normalization, grading utilities
• stage1_filter → stage4_judge – sequential scripts to filter, format, merge, and judge examples
• training/
• verl/ – extension of the veRL framework with model loaders, controllers (Ray, Megatron, vLLM), distributed settings, trainer entrypoints (main_sft, main_ppo, main_eval)
• scripts & examples – data_prepare, run_prime_main.sh
• requirements & setup
• eval/
• Coding/ – subfolders for human_eval, LeetCode, LiveCodeBench runner
• Math/ – Qwen25‐Math evaluation with LaTeX→Sympy parser & grader, plus AMC, AIME, etc.
• utils/ – shared evaluation, execution, grading, data loading
Detail
• Configuration files ➔ controller initializations
• Versioned model repos on Hugging Face
• Scripts (run.sh, main_ppo.py, evaluate_leetcode.py) at the edges as entrypoints