How damanimehul/rlcr works
Project Type and Purpose
• Project type: Research‐oriented ML pipeline implementing reinforcement learning for large‐language models.
• Purpose: Train LMs with uncertainty‐aware reward signals (Beyond Binary Rewards) and evaluate them across QA/math datasets.
File Structure
• Top‐level scripts:
– rl_runner.py, GRPO_Trainer.py: Orchestrate training with TRL + deepspeed/accelerate
– evaluation.py, inference_example.py: Run batch evaluation and demo inference
– train_runs.sh, eval_runs.sh: Convenience wrappers
– dataset_processing.py: Dataset loading & preprocessing
– reward_fns.py: Custom reward‐computation functions
• Directories:
– configs/Qwen-7B/*: YAML training configs (datasets, RLCR/RLVR variants)
– data/*: Raw & processed dataset splits + creation scripts
– eval/: Core evaluation functions & utils
– eval_configs/*: JSON eval scenario definitions
– eval_outputs/ & results/: Persisted eval outputs & metrics
• Config/build files: deepspeed.yaml, environment.yml
Extract README Insights
• Key external dependencies: HuggingFace TRL, Open-R1, deepspeed, accelerate, wandb
• Training flow: accelerate launch → rl_runner → distributed RL training on multi‐GPU
• Evaluation flow: evaluation.py / eval_runs.sh → generate predictions → compute metrics
Main Components & Their Interactions
a. Configuration Manager
– Inputs: YAML (configs/) for training, JSON (eval_configs/) for evaluation
– Provides hyperparameters, model paths, dataset identifiers
b. Data Ingestion & Preprocessing
– Supplies batched examples to the training or evaluation pipeline
c. Training Orchestrator
– rl_runner.py and GRPO_Trainer.py wrap HuggingFace TRL RL loops
– Leverages reward_fns.py for computing sample‐level rewards
– Uses deepspeed.yaml + accelerate for distributed execution
– Logs metrics and intermediate generations to wandb
d. Reward Module
– reward_fns.py encapsulates “Beyond Binary” reward logic
– Called by the RL loop to score model outputs against ground truth
e. Model Artifact Manager
– Checkpoints saved locally or to HuggingFace hub
– Config files (configs/Qwen-7B/…) track experiment provenance
f. Evaluation Pipeline
– evaluation.py invokes eval/check_functions.py, eval_utils.py
– Consumes eval_configs/*, loads trained model, runs inference
– Writes raw outputs to eval_outputs/ and summary metrics to results/
g. Inference Demo
– inference_example.py: spin up single‐GPU run for ad‐hoc testing
h. Auxiliary Scripts
– train_runs.sh / eval_runs.sh: Shell scripts combining steps above
Architectural Patterns & Principles
• Modular ML‐pipeline: Clear separation of “Config → Data → Training → Reward → Logging → Evaluation”
• Plugin‐style Reward & Eval: New datasets or reward strategies can be added by dropping files under data/creation_scripts/, reward_fns.py, or eval_configs/
• Distributed Compute Abstraction: Deepspeed + Accelerate hide multi‐GPU details
• CLI‐driven Orchestration: Single‐entrypoints (rl_runner.py, evaluation.py, inference_example.py)
How to Represent This
– Configuration (YAML/JSON)
– Data storage (data/, eval_outputs/)
– Compute modules (Training, Reward, Evaluation)
– External services (wandb, HuggingFace Model Hub, GPUs)
– Config → Runner → Data Loader → RL Loop → Reward Module → Checkpoints → Evaluation → Results
– Runner → wandb API calls
• Critical dependencies:
– TRL & Open-R1 libraries under “RL Loop”
– deepspeed & accelerate under “Distributed Compute”
– datasets & transformers under “Data Loader” and “Model”