# om-ai-lab/vlm-r1 architecture

> Architecture diagram of the GitHub repository om-ai-lab/vlm-r1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/om-ai-lab/vlm-r1
- Repository: https://github.com/om-ai-lab/vlm-r1

## Overview

##### Main Components Identification
- Core Training Module: The heart of the system lies in the “src/open-r1-multimodal” directory. It includes:
– Setup scripts (setup.sh, Dockerfile)
– Configuration files (YAML configs in “configs” and “data_config” folders)
– Model code under “src/open_r1” with sub-modules for evaluation, generation, different GRPO variants, SFT, and additional trainer modules.
- Data Handling & Preprocessing: There are directories and scripts for data management:
– “src/open-r1-multimodal/data_config” for dataset configurations.
– “src/open-r1-multimodal/data_jsonl” and dedicated scripts like “grpo_jsonl.py” to handle custom JSONL inputs and multi-image input formats.
- Run & Training Scripts: Located in “src/open-r1-multimodal/run_scripts” and “local_scripts”, these take care of launching different training modes (e.g., single-node, multi-node, GUI-based evaluation) as well as fine-tuning strategies.
- Evaluation Module: Found in “src/eval”—includes evaluation scripts (e.g., test_rec_r1.py and test_rec_baseline.py) and setups for measuring performance.

##### Relationships and Interactions
- The training scripts and run scripts (inside “run_scripts” and “local_scripts”) interact with core modules (in “src/open_r1” and “trainer”) by passing configuration parameters (from YAML and JSON config files).
- Data flows from external sources (images and JSON/JSONL annotation files) into data ingestion routines (as configured by “data_config/rec.yaml”) and then gets processed by the GRPO or SFT training pipelines.
- The evaluation scripts in “src/eval” depend on the output models generated by the training process and feed into external performance reporting tools (e.g., WandB, as hinted in the training commands).

##### Architectural Patterns and Design Principles
- Modular Structure: The code base is split into clearly delineated modules (e.g., model configuration, training, utilities, and evaluation), which helps isolate concerns.
- Configuration-driven Execution: Extensive use of YAML files for model, training, and data configuration promotes flexibility and dynamic adjustment.
- Support for Distributed Training: The use of “torchrun”, multi-node training scripts, and Deepspeed configuration (e.g., through “zero2.yaml”, “zero3.yaml” scripts) indicates a design aimed at scaling across multiple GPUs/nodes.
- Fine-tuning Approaches: Separation between full fine-tuning (GRPO) and parameter-efficient fine-tuning (LoRA, SFT via LLaMA-Factory) points to extensible design for experimental research in model training.

##### Relevant Technologies, Frameworks, or Libraries
- Python (with a clear version requirement – Python 3.10)
- Deep learning frameworks: PyTorch, Deepspeed, and torchrun for distributed training.
- Docker for containerized deployment.
- Additional integration with external libraries such as WandB for logging and monitoring.
- LLaMA-Factory integration for SFT model training.

##### Overall Detailed Approach
- Nodes for auxiliary scripts (such as multi-node training scripts and local scripts for creating specific training data or launching experiments).
- Don’t oversimplify—detail each component such that interactions like model generation, evaluation callbacks, and configuration dependencies are clearly visible.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Infrastructure Setup"
        SetupAndEnv("Setup & Env Config"):::env
        Config("Configuration Manager"):::config
    end

    subgraph "Data Layer"
        ExternalData("External Data Sources (Images/Annotations)"):::external
        DataIngest("Data Ingestion & Preprocessing"):::data
    end

    subgraph "Training Pipeline"
        TrainingScripts("Run & Training Scripts"):::script
        CoreTraining("Core Training Module"):::core
        subgraph "Fine-Tuning Strategies"
            GRPO("GRPO Training"):::core
            LoRA("LoRA Fine-tuning"):::core
            SFT("SFT (LLaMA-Factory)"):::core
        end
    end

    subgraph "Evaluation & Monitoring"
        Eval("Evaluation Module"):::eval
        Monitoring("External Monitoring (WandB)"):::external
    end

    %% Data and Control Flow
    SetupAndEnv --> Config
    ExternalData --> DataIngest
    Config --> DataIngest
    Config --> CoreTraining
    DataIngest --> CoreTraining
    TrainingScripts --> CoreTraining
    CoreTraining --> GRPO
    CoreTraining --> LoRA
    CoreTraining --> SFT
    GRPO --> Eval
    LoRA --> Eval
    SFT --> Eval
    Eval --> Monitoring

    %% Click Events for Component Mapping
    click CoreTraining "https://github.com/om-ai-lab/vlm-r1/tree/main/src/open-r1-multimodal/src/open_r1"
    click SetupAndEnv "https://github.com/om-ai-lab/vlm-r1/blob/main/setup.sh"
    click SetupAndEnv "https://github.com/om-ai-lab/vlm-r1/tree/main/Dockerfile"
    click Config "https://github.com/om-ai-lab/vlm-r1/tree/main/src/open-r1-multimodal/configs"
    click Config "https://github.com/om-ai-lab/vlm-r1/tree/main/src/open-r1-multimodal/data_config"
    click DataIngest "https://github.com/om-ai-lab/vlm-r1/tree/main/src/open-r1-multimodal/data_jsonl"
    click DataIngest "https://github.com/om-ai-lab/vlm-r1/blob/main/src/open-r1-multimodal/src/open_r1/grpo_jsonl.py"
    click TrainingScripts "https://github.com/om-ai-lab/vlm-r1/tree/main/src/open-r1-multimodal/run_scripts"
    click TrainingScripts "https://github.com/om-ai-lab/vlm-r1/tree/main/src/open-r1-multimodal/local_scripts"
    click Eval "https://github.com/om-ai-lab/vlm-r1/tree/main/src/eval"

    %% Style Definitions
    classDef env fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef config fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef data fill:#A3E4D7,stroke:#333,stroke-width:2px;
    classDef core fill:#F3F781,stroke:#333,stroke-width:2px;
    classDef script fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef eval fill:#BB8FCE,stroke:#333,stroke-width:2px;
    classDef external fill:#82E0AA,stroke:#333,stroke-width:2px;
```
