How nick7nlp/fastcurl works
Main Components
• Data Layer: Represent the training and test data stored as Parquet files (located under fastcurl/fastcurl/data/train and fastcurl/fastcurl/data/test). This layer is the source of problem-answer pairs compiled from various datasets.
• Core FastCuRL Module: This is in the fastcurl/fastcurl directory and encapsulates:
– global configuration (globals.py)
– utilities (utils.py)
– system prompts (system_prompts.py)
– reward computation modules (in rewards/ – includes math_reward and associated utilities)
• Training Orchestration & Scripts:
– The shell scripts in fastcurl/scripts/train (for each stage with different context lengths: 8K, 16K, 24K) initiate the training routines.
– The evaluation scripts under fastcurl/scripts/eval further rally the evaluation process.
• Distributed Training & Model Management:
– The fastcurl/verl directory is the heavily modified fork of “verl” which provides distributed orchestration. It contains:
▪ Model Definitions and Loaders (under verl/models – supporting multiple architectures such as LLaMA, Megatron, and Transformer-based models)
▪ Single controller / distributed communication modules (under verl/single_controller with submodules for Ray and Megatron)
▪ Trainer modules and configuration files (under verl/trainer which include YAML configuration for evaluation, generation, and PPO training)
▪ Utilities for distributed training, logging, debugging, and performance monitoring (under verl/utils)
▪ Third-party integrations with vLLM (under verl/third_party) which further facilitate efficient inference and weight loading.
Relationships and Interactions
• The training/evaluation scripts act as “entry points” that call into the core FastCuRL library for data handling and reward computation.
• The FastCuRL main module (fastcurl/fastcurl) provides the logic for reinforcement learning training, including processing prompts, applying reward signals, and managing curriculum stages.
• The “verl” component wraps the distributed model handling and training strategies:
– Training scripts invoke modules from verl/trainer.
– The distributed controller (from single_controller) schedules tasks across GPUs using Ray or Megatron, as seen in scripts and configuration files.
– Model checkpoint loading/saving and parallel executions are managed by components in verl/models and the third-party wrappers in vLLM directories.
• Data flows from the raw parquet datasets into the training pipeline, then through the model architecture where rewards (calculated by modules in fastcurl/fastcurl/rewards) are fed back to update model parameters iteratively.
• Configuration files (YAMLs) in verl/trainer/config help parameterize various training aspects, including evaluation metrics and generation settings.
Architectural Patterns and Design Principles
• Modularity: The system is highly modular with clear separation between data handling, reward computation, and distributed training/model management.
• Distributed / Parallel Training: Use of frameworks like Ray, Megatron, and vLLM that enable scaling training across multiple GPUs. This is evident in the distributed controller modules and scripts.
• Curriculum-based Reinforcement Learning: Training happens in multiple stages (evident from the four training stage scripts, each increasing the context length), making use of a progressive data presentation strategy.
• Reusability and extensibility: The separation into “fastcurl” (core functionality) and “verl” (extended distributed training and model components) suggests an architecture designed for reuse and further integration.
Elements
• Component Labels: Name each block (e.g., “Data Repository”, “FastCuRL Core”, “Reward Modules”, “Training Scripts”, “Distributed Controller (Ray/Megatron)”, “Model Repository”).
– From the “Data Repository” to "FastCuRL Core" (data ingestion).
– From "FastCuRL Core" to “Training Scripts” (training logic).
– From “Training Scripts” to “Distributed Training Modules” (orchestration across GPUs).
• Notation for Stages: Optionally, indicate the four training stages (Stage I: 8K, Stage II: 16K, Stage III: 24K, and Stage IV: 24K) near the training script section to capture the curriculum components.
Final Recommendations
• Be very detailed: Separate the project into as many logical components as possible to capture the nuances of its architecture.