# internlm/caprl architecture

> Architecture diagram of the GitHub repository internlm/caprl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internlm/caprl
- Repository: https://github.com/internlm/caprl

## Overview

##### Project type and purpose
- This is an ML research codebase for training, evaluating, and serving an image-captioning model called CapRL.
- Components include:
- QA data curation pipeline (generate, extract, answer, filter QA)
- Pretraining/training recipe (initializing Qwen2.5-VL, running RL training via LLaMAFactory)
- Evaluation framework (Prism_Evaluation) to assess caption quality by downstream VQA
- Inference/serving scripts (vLLM-based OpenAI-compatible API)

##### File structure
- Top-level:
- QA_data_curation/ – four sequential subfolders (1_generate_qa, 2_extract_qa, 3_answer_qa, 4_filter_qa)
- Prism_Evaluation/ – evaluation scripts (Eval_CapRL.py, reward-model wrappers)
- eval_results/ – JSON results of Prism runs
- assets/ – images used in README and performance figures
- setup.sh, pyproject.toml – environment setup and dependencies
- Shell scripts (bash.sh, rjob.sh) launch jobs on clusters or vLLM service
- No traditional “frontend/backend” split; instead a data-science pipeline and service component

##### Key insights from README
- QA curation consists of four distinct stages for building a filtered 75K QA dataset
- Pretraining uses Qwen2.5-VL-3B as an initial policy, with LLaMAFactory for RL
- Inference is served via vLLM, exposing an OpenAI-compatible API
- Evaluation uses the Prism framework: generate caption → ask VQA model → compute reward

##### Detail
- Shell scripts as triggers at each stage
- Parameters (e.g., ROTATE_NUM, reward-model-path)
- RL training recipe and HuggingFace components

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage
    subgraph "Data Storage"
        DS1["Raw QA Sources (ShareGPT-1M, DenseFusion-1M)"]:::data
        DS2["Assets/Images"]:::data
    end

    %% QA Data Curation Pipeline
    subgraph "QA Data Curation Pipeline"
        QC1["Step1: QA Generation\n(gen_qa.py)"]:::compute
        QC1_sh["Trigger: gen.sh / rjob.sh"]:::compute
        QC2["Step2: QA Extraction\n(extract_qa.ipynb)"]:::compute
        QC3["Step3: QA Answering\n(answer_with_without_image.py)"]:::compute
        QC3_sh["Trigger: launch.sh / rjob.sh"]:::compute
        QC4["Step4: QA Filtering\n(filter_qa.ipynb)"]:::compute
        QC_filtered["Filtered QA Dataset"]:::data
    end

    %% Pretraining Pipeline
    subgraph "Pretraining & RL Training"
        PT1["Initialize Qwen2.5-VL-3B\n(initiallize_vlm_3b.ipynb)"]:::compute
        PT2["RL Training\n(LLaMAFactory)"]:::compute
    end

    %% Model Repository
    MR["Model Repository\n(CapRL-3B, InternVL3.5-8B, Eval-3B)"]:::storage

    %% Inference Service
    IS["Inference Service\n(vLLM OpenAI-compatible API)"]:::service

    %% Evaluation Pipeline
    subgraph "Evaluation (Prism_Evaluation)"
        EV1["Eval_CapRL.py"]:::compute
        EV2["vqa_eval.py & vlm_misc.py"]:::compute
        EV_output["eval_results (JSON)"]:::data
    end

    %% Job Orchestration / Setup
    JO1["setup.sh / pyproject.toml"]:::compute

    %% Client
    Client["Client Applications"]:::service

    %% Relationships
    DS1 -->|provides| QC1
    DS2 -->|images| QC1
    QC1 --> QC1_sh
    QC1 --> QC2
    QC2 --> QC3
    QC3 --> QC3_sh
    QC3 --> QC4
    QC4 --> QC_filtered
    QC_filtered --> PT1
    PT1 --> PT2
    PT2 --> MR
    MR --> IS
    IS --> Client
    MR --> EV1
    QC_filtered --> EV1
    EV1 --> EV2
    EV2 --> EV_output
    JO1 --> QC1_sh
    JO1 --> QC3_sh
    JO1 --> EV2

    %% Click Events - Data Storage
    click DS2 "https://github.com/internlm/caprl/tree/main/assets/"

    %% Click Events - QA Data Curation
    click QC1 "https://github.com/internlm/caprl/blob/main/QA_data_curation/1_generate_qa/gen_qa.py"
    click QC1_sh "https://github.com/internlm/caprl/blob/main/QA_data_curation/1_generate_qa/gen.sh"
    click QC1_sh "https://github.com/internlm/caprl/blob/main/QA_data_curation/1_generate_qa/rjob.sh"
    click QC1 "https://github.com/internlm/caprl/blob/main/QA_data_curation/1_generate_qa/1_input_sample.json"
    click QC2 "https://github.com/internlm/caprl/blob/main/QA_data_curation/2_extract_qa/extract_qa.ipynb"
    click QC2 "https://github.com/internlm/caprl/blob/main/QA_data_curation/2_extract_qa/2_output_sample.json"
    click QC3 "https://github.com/internlm/caprl/blob/main/QA_data_curation/3_answer_qa/answer_with_without_image.py"
    click QC3_sh "https://github.com/internlm/caprl/blob/main/QA_data_curation/3_answer_qa/launch.sh"
    click QC3_sh "https://github.com/internlm/caprl/blob/main/QA_data_curation/3_answer_qa/rjob.sh"
    click QC4 "https://github.com/internlm/caprl/blob/main/QA_data_curation/4_filter_qa/filter_qa.ipynb"

    %% Click Events - Evaluation
    click EV1 "https://github.com/internlm/caprl/blob/main/Prism_Evaluation/Eval_CapRL.py"
    click EV2 "https://github.com/internlm/caprl/blob/main/Prism_Evaluation/openrlhf/models/remote_rm/vqa_eval.py"
    click EV2 "https://github.com/internlm/caprl/blob/main/Prism_Evaluation/openrlhf/models/remote_rm/vlm_misc.py"
    click EV2 "https://github.com/internlm/caprl/blob/main/Prism_Evaluation/scripts/bash.sh"
    click EV2 "https://github.com/internlm/caprl/blob/main/Prism_Evaluation/scripts/rjob.sh"

    %% Click Events - Setup
    click JO1 "https://github.com/internlm/caprl/blob/main/setup.sh"
    click JO1 "https://github.com/internlm/caprl/blob/main/pyproject.toml"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef compute fill:#D4FFD4,stroke:#333,stroke-width:1px
    classDef storage fill:#FFE4B3,stroke:#333,stroke-width:1px
    classDef service fill:#FFD4D4,stroke:#333,stroke-width:1px
```
