# opengvlab/internvl architecture

> Architecture diagram of the GitHub repository opengvlab/internvl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opengvlab/internvl
- Repository: https://github.com/opengvlab/internvl

## Overview

- Data/Configuration Layer:
– Include directories such as “classification/configs,” “segmentation/configs,” and “meta_data.”
– These files (YAML, JSON, txt.zip) serve as the configuration for experiments (e.g., probing, linear probing, semantic segmentation setups) and model hyperparameters.
- Data Processing & Dataset Module:
– The “classification/dataset” folder and similar support code for loading, caching, and sampling (e.g., “zipreader.py”, “samplers.py”).
- Model Building & Training Modules:
– “classification/models,” “segmentation/mmseg_custom,” “internvl_chat/internvl/model,” “internvl_g/internvl/model” all have distinct implementations.
– Include utilities like “build.py,” “optimizer.py,” “lr_scheduler.py,” and patch files used for distributed training.
- Evaluation & Benchmarking Modules:
– “clip_benchmark” and evaluation scripts for classification, segmentation, image–text retrieval, video retrieval.
– Benchmarks for zero‑shot classification, retrieval, and metric evaluation (e.g., “metrics” subfolders in clip_benchmark).
- Multimodal Chat/Dialogue Module:
– “internvl_chat” (and “internvl_chat_llava”), “internvl_g” folders hold conversation models and training scripts along with shell scripts.
- Demo & Serving/Deployment Layer:
– “streamlit_demo” contains API, controller, and front‑end code for running interactive demos.
- Utility & Infrastructure Components:
– General purpose utilities (e.g., “utils.py”), logging (“logger.py”), pre‑commit hooks (flake8, isort, etc.), and deployment scripts.
– Build scripts and configuration files (pyproject.toml, .pre‑commit‑config.yaml, etc.) for integration and deployment.

##### How the Components Inter-Interact
- Data flows start in the “Data/Configuration” layer, where YAML/JSON files are used to set up experiments.
- The “Dataset” modules read raw image/text data, pre‐process it (sometimes with dedicated utilities like “cached_image_folder.py” and “zipreader.py”), then feed it into model training pipelines.
- The “Model Building” modules use the configuration, build architectures (e.g., InternViT, InternVL Chat modules, InternVL-G/C replacements), and then pass data through training pipelines (with distributed support via “ddp_hooks.py”).
- Once trained, models are evaluated by the “Benchmark/Evaluation” modules. For example, the clip_benchmark and segmentation evaluation scripts receive models from the training modules and process standardized datasets.
- For multimodal dialogue, the “Chat” modules (in internvl_chat) accept both text and image/video inputs, using inference routines that delegate to the multimodal encoders (from the classification/segmentation modules).
- Finally, demo components (like Streamlit) interact with these trained models by calling APIs (potentially wrapping Hugging Face’s AutoModel and AutoTokenizer interfaces) and then displaying outputs.

##### Architectural Patterns and Design Principles
- Modularity & Separation of Concerns: Each major task (classification, segmentation, chat) is separated into its own directory and codebase. This modular design enables separate development, testing, and deployment of components.
- Configuration-Driven Design: The extensive use of YAML/JSON configurations indicates that experiments and model variants are data‑driven and easier to reproduce.
- Reuse & Extensibility: Similar submodules (e.g., “build.py” files and patch scripts) are reused across different tasks; they form the common backbone (internally built atop popular frameworks such as PyTorch and Transformers).
- Distributed Training: File names such as “ddp_hooks.py” and optimizer constructors show that the architecture supports scaling through multi‑GPU/distributed training.
- Plug-and-Play Pipeline: The unified usage of Hugging Face Transformers’ interfaces in the quick‑start examples shows that the front‑end demos and inference routines are designed to “plug and play” with components.

##### Incorporate Technologies, Frameworks, and External Systems
- Base Frameworks: PyTorch for model training and inference, Hugging Face Transformers for interoperability.
- Data Processing: Libraries like Decord (video), Pillow (image), and TorchVision transforms.
- Deployment & Demo: Streamlit for web-based UI; several shell scripts for distributed training and deployment.

- Keep it high‑level yet detailed: ensure you capture all key components and data flows without oversimplifying complex interdependencies.
- Versioning and model family differences (for InternVL 1.x, 2.0, 2.5) as “sub‑components” if needed.
- A layered approach where the bottom layer is “Infrastructure” (configuration, pre‑commit hooks, GitHub integration) and the top layer is “User Interface/Deployment.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Utility & Infrastructure Components
    subgraph "Utility & Infrastructure"
        UT1("General Utilities"):::util
        UT2("Logger"):::util
        UT3("Chat Patch Utils"):::util
        UT4("Build Scripts & Configs"):::util
        UT5("Pre-commit Config"):::util
    end

    %% Data/Configuration Layer
    subgraph "Data/Configuration Layer"
        DC1("Classification Configs"):::cfg
        DC2("Segmentation Configs"):::cfg
        DC3("Meta Data"):::cfg
    end

    %% Data Processing & Dataset Module
    subgraph "Data Processing & Dataset Module"
        DP1("Classification Dataset"):::proc
        DP2("Segmentation Datasets"):::proc
    end

    %% Model Building & Training Modules
    subgraph "Model Building & Training Modules"
        MT1("Classification Models"):::train
        MT2("Segmentation Models"):::train
        MT3("InternVL Chat Models"):::train
        MT4("InternVL G Models"):::train
        MT5A("DDP Hooks"):::train
        MT5B("Optimizer"):::train
        MT5C("LR Scheduler"):::train
    end

    %% Evaluation & Benchmarking Modules
    subgraph "Evaluation & Benchmarking Modules"
        EV1("CLIP Benchmark"):::eval
        EV2("InternVL Chat Evaluation"):::eval
        EV3("InternVL G Evaluation"):::eval
    end

    %% Multimodal Chat/Dialogue Module
    subgraph "Multimodal Chat/Dialogue Module"
        CH1("InternVL Chat"):::chat
        CH2("InternVL Chat LLaVA"):::chat
    end

    %% Demo & Serving/Deployment Layer
    subgraph "Demo & Serving/Deployment Layer"
        DS1("Streamlit Demo"):::demo
    end

    %% Connections from Data/Configuration to Data Processing
    DC1 -- "influences" --> DP1
    DC2 -- "influences" --> DP2
    DC3 -- "influences" --> DP1
    DC3 -- "influences" --> DP2

    %% Connections from Data Processing to Model Building
    DP1 -- "feeds" --> MT1
    DP2 -- "feeds" --> MT2

    %% Connections from Common Training Utilities to Model Building Modules
    MT5A -- "supports" --> MT1
    MT5A -- "supports" --> MT2
    MT5A -- "supports" --> MT3
    MT5A -- "supports" --> MT4

    MT5B -- "supports" --> MT1
    MT5B -- "supports" --> MT2
    MT5B -- "supports" --> MT3
    MT5B -- "supports" --> MT4

    MT5C -- "supports" --> MT1
    MT5C -- "supports" --> MT2
    MT5C -- "supports" --> MT3
    MT5C -- "supports" --> MT4

    %% Connections from Model Building to Evaluation
    MT1 -- "evaluated_by" --> EV1
    MT2 -- "evaluated_by" --> EV1
    MT3 -- "evaluated_by" --> EV2
    MT4 -- "evaluated_by" --> EV3

    %% Connections from Model Building to Chat Module
    MT3 -- "deploys_to" --> CH1

    %% Connections from Evaluation & Chat Modules to Demo
    EV1 -- "inference_for" --> DS1
    CH1 -- "inference_for" --> DS1

    %% Utility & Infrastructure dotted support connections
    UT4 --- DC1
    UT4 --- DC2
    UT5 --- DC1
    UT1 --- MT1
    UT2 --- MT1
    UT3 --- CH1

    %% Click Events
    click DC1 "https://github.com/opengvlab/internvl/tree/main/classification/configs"
    click DC2 "https://github.com/opengvlab/internvl/tree/main/segmentation/configs"
    click DC3 "https://github.com/opengvlab/internvl/tree/main/classification/meta_data"
    click DP1 "https://github.com/opengvlab/internvl/tree/main/classification/dataset"
    click DP2 "https://github.com/opengvlab/internvl/tree/main/segmentation/configs/_base_/datasets"
    click MT1 "https://github.com/opengvlab/internvl/tree/main/classification/models"
    click MT2 "https://github.com/opengvlab/internvl/tree/main/segmentation/mmseg_custom"
    click MT3 "https://github.com/opengvlab/internvl/tree/main/internvl_chat/internvl/model"
    click MT4 "https://github.com/opengvlab/internvl/tree/main/internvl_g/internvl/model"
    click MT5A "https://github.com/opengvlab/internvl/blob/main/classification/ddp_hooks.py"
    click MT5B "https://github.com/opengvlab/internvl/blob/main/classification/optimizer.py"
    click MT5C "https://github.com/opengvlab/internvl/blob/main/classification/lr_scheduler.py"
    click EV1 "https://github.com/opengvlab/internvl/tree/main/clip_benchmark"
    click EV2 "https://github.com/opengvlab/internvl/tree/main/internvl_chat/eval"
    click EV3 "https://github.com/opengvlab/internvl/tree/main/internvl_g/eval"
    click CH1 "https://github.com/opengvlab/internvl/tree/main/internvl_chat"
    click CH2 "https://github.com/opengvlab/internvl/tree/main/internvl_chat_llava"
    click DS1 "https://github.com/opengvlab/internvl/tree/main/streamlit_demo"
    click UT1 "https://github.com/opengvlab/internvl/blob/main/classification/utils.py"
    click UT2 "https://github.com/opengvlab/internvl/blob/main/classification/logger.py"
    click UT3 "https://github.com/opengvlab/internvl/tree/main/internvl_chat/internvl/patch"
    click UT4 "https://github.com/opengvlab/internvl/blob/main/pyproject.toml"
    click UT5 "https://github.com/opengvlab/internvl/blob/main/.pre-commit-config.yaml"

    %% Styles
    classDef cfg fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef proc fill:#d4edda,stroke:#333,stroke-width:2px;
    classDef train fill:#f8d7da,stroke:#333,stroke-width:2px,stroke-dasharray: 5, 5;
    classDef eval fill:#fff3cd,stroke:#333,stroke-width:2px;
    classDef chat fill:#d1c4e9,stroke:#333,stroke-width:2px;
    classDef demo fill:#c8e6c9,stroke:#333,stroke-width:2px;
    classDef util fill:#eeeeee,stroke:#333,stroke-width:2px,stroke-dasharray: 2, 2;
```
