How deep-agent/r1-v works
Project Type and Purpose
• The project is a research-oriented open‐source framework for enhancing Vision Language Models (VLMs) through reinforcement learning and supervised fine-tuning.
• Its purpose is to experiment with and improve general visual reasoning (e.g., counting and geometry reasoning) on established datasets (CLEVR, GEOQA) using methods like GRPO and SFT.
Analyzing the File Structure
• Top-level files/directories:
– .gitignore, README.md, setup.sh – for basic project configuration.
– images – for assets used in the documentation.
– src directory – which contains the main source code.
• Within src, there are several key submodules:
a. src/distill_r1:
• Contains scripts and notebooks (e.g., generate_scene_qa_pairs.ipynb, prompt.py, query_r1.py) designed for data distillation and dataset creation.
b. src/eval:
• Houses evaluation scripts, logs, and prompt files. This module is clearly dedicated to assessing the trained models on tasks such as SuperCLEVR and GEOQA.
c. src/r1-v:
• This is the core training and experimentation engine. It includes:
▪ Configuration files (in the configs directory) for different training setups (ddp, zero configs, qwen2vl_sft_config).
▪ Local scripts to generate data, prepare training, and run experiments.
▪ Main modules in src/open_r1 with files for evaluation (evaluate.py), generation (generate.py), reinforcement learning policy optimization (grpo.py), and supervised fine-tuning (sft.py).
▪ Trainer implementations including GRPO and modified VLLM-based trainers in the trainer subdirectory.
d. src/scripts:
• Contains convenient shell scripts to execute various experiments (e.g., running GRPO with different configurations, SFT, tests for specific datasets).
Extracting Insights from the README
• The README emphasizes two major training paradigms: GRPO (reinforcement learning) and SFT (supervised fine tuning).
• It lists supported models (Qwen2-VL, Qwen2.5-VL) and details datasets including CLEVR-70k and GEOQA-8k.
• It provides command-line examples with torchrun (distributed training) and accelerate (for SFT), which suggests distributed setups and GPU/multi-node usage.
• External resources (e.g., HuggingFace datasets, Git LFS for large images) also play a role.
a. Main Components
• Data Preparation & Distillation Module:
▪ Represented by src/distill_r1.
▪ Responsible for dataset generation, filtering, and distillation.
• Core Training Engine:
▪ Represented by src/r1-v.
▪ Contains submodules for RL-based training (GRPO) and SFT.
▪ Includes configurations, local scripts, and trainer implementations.
• Evaluation & Testing Module:
▪ Represented by src/eval and related scripts.
▪ Handles testing (SuperCLEVR and GEOQA), log generation, and result analysis.
• Utility and Support Scripts:
▪ Represented by setup.sh, src/scripts, and configuration files.
▪ They handle environment setup, command orchestration, and deployment scripts.
• External Integration:
▪ Datasets from HuggingFace, external images (using Git LFS), and possibly logging/visualization platforms like WandB.
• Distributed/Parallel Training Infrastructure:
▪ Incorporate torchrun, accelerate, DeepSpeed, and vLLM support that tie into the core training engine.
b. Relationships & Interactions
• Data flows from the Data Preparation Module to the Core Training Engine.
• The Core Training Engine outputs models trained via either GRPO or SFT.
• The Evaluation Module takes the trained models as input, runs tests (counting, geometry reasoning), and outputs logs/results.
• Utility scripts and configuration files interface with both training and evaluation pipelines.
• External datasets and resources are fed into the Data Preparation Module and referenced by training/evaluation scripts.
c. Architectural Patterns & Design Principles
• Separation of Concerns:
▪ Data distillation, training, and evaluation are managed in their own subdirectories.
• Modular Design:
▪ Common configurations and shared modules (such as open_r1) that are reused across different experiments.
• Distributed & Scalable Training:
▪ Use of distributed training commands (torchrun, DeepSpeed, vLLM) to support multi-GPU setups.
• Experimentation & Extensibility:
▪ Use of separate local scripts and configuration files that make it straightforward to plug in new datasets or training methods.
d. Relevant Technologies, Frameworks, or Libraries
• Python 3.11 (as per setup.sh and conda instructions)
• PyTorch (distributed training via torchrun)
• DeepSpeed (as shown by the JSON config files)
• vLLM and accelerate for optimized training and GPU utilization
• Jupyter Notebook (as indicated by *.ipynb files for dataset generation)
• Git LFS for managing large data (images)
• WandB for logging experiments
Elements
• From External Data Sources → Data Distillation.
• Data Distillation → Training Engine.
• Training Engine → Evaluation.
• Differentiate between computation nodes (training modules), data repositories (datasets, logs), and orchestration scripts.