How agentica-project/rllm works
Based on our analysis of the file tree and README, this project is a reinforcement learning (RL) framework for fine-tuning large language models (LLMs) applied to both coding and mathematical tasks. It provides end‐to‐end support from data preprocessing (for code and math datasets) and reward functions (for both domains), to multi-node RL training and evaluation. In other words, it’s an open-source toolkit for democratizing RL for LLMs, integrating a variety of modules from data handling to distributed training engines (using, for example, Ray) and specialized pipelines (like the verl-pipe extension).
• Data Layer:
• Raw Data & Preprocessing: Represent the directories under rllm/data (with subfolders train, test, and preprocess for code and math). Note that there are individual preprocessing notebooks and scripts (e.g., in rllm/data/preprocess/code and rllm/data/preprocess/math) that convert raw JSON to Parquet files.
• Reward Functions:
• Rewards module (rllm/rewards) which includes separate reward functions for code (e.g., code_reward) and math (math_reward) as well as supporting utilities.
• Core Library & Utilities:
• Main package (rllm) with global settings (globals.py), utility functions (utils.py), and system prompts. This forms the central “engine” used by training, evaluation, and tools.
• Tools & Extensions:
• Training & Evaluation Pipelines:
• Training scripts (inside scripts/deepcoder and scripts/deepscaler) and the evaluation scripts (scripts/eval). These scripts orchestrate both single-node and multi-node (Ray-based) training and evaluation.
• Distributed & Parallel Training/System Integration:
• Integration with the verl modules (and its pipeline enhancement “verl-pipe”) which seem to extend the core RL training on top of an adapted version of an RLHF library.
• Multi-node training setup that uses Ray (with instructions for head node and worker nodes). This represents how the training tasks are scaled.
• Testing & Examples:
• Don’t forget to include a component representing the tests (in tests/ and in verl/tests) that validate the reward computations, data pipelines, and model generation.
• External Dependencies & Integrations:
• References to external systems such as Hugging Face (for model hosting and datasets), Wandb (for training logs), and cloud storage (Google Drive for evaluation logs).
Relationships & Interactions
• Data Flow:
• How raw datasets (rllm/data/train and rllm/data/test) are processed through the preprocessing notebooks/scripts.
• That processed data feeds into the training pipeline.
• Reward Signal Path:
• How the preprocessed data and model outputs are evaluated by reward functions from rllm/rewards. The resulting reward signals are then looped back to influence the reinforcement learning training.
• Training & Evaluation Flow:
• Control flow from training scripts (in scripts/*/train) which interact with the core library, distributed training (Ray and verl), and finally the evaluation module that uses evaluation scripts.
• Distributed Coordination:
• Pipeline Acceleration:
Architectural Patterns & Design Principles
• Modular Separation:
• Distributed & Parallel Processing:
• Components such as Ray clusters, asynchronous pipelines, and multi-tool integrations that are central to scaling the RL training.
• Reusability & Extensibility:
• “tools” layer that provides specialized functions and integrations (e.g., code evaluation, math calculation, web searches) that can be extended or swapped out.
• Clear Interface Boundaries:
• Interfaces between data preprocessing, reward functions, training scripts, and evaluation. This delineates responsibilities and dependencies in the architecture.
Detail the Data & Control Flow
• Demonstrate the preprocessing step leading to standardized Parquet format.
• How the training scripts call the core rllm functions (incorporating reward functions and system prompts).
• Distributed communication layer (Ray cluster and verl modules) orchestrating multi-node training.