# open-compass/vlmevalkit architecture

> Architecture diagram of the GitHub repository open-compass/vlmevalkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/open-compass/vlmevalkit
- Repository: https://github.com/open-compass/vlmevalkit

## Overview

##### Project type and purpose
- VLMEvalKit is an **open-source evaluation toolkit** (Python package `vlmeval`) for **large vision-language models** (LVLMs). It provides a one-command evaluation pipeline over multiple benchmarks (image and video) by orchestrating data loading, model inference (via APIs or local calls), and metric computations.

##### File structure
- Top-level entrypoint:
- `run.py` – CLI for kicking off evaluations, uses a config system (`vlmeval/config.py`).
- `setup.py` / `requirements.txt` – installation and dependencies (transformers, torch, etc.).
- Core modules under `vlmeval/`:
- `api/` – wrappers for remote/commercial model APIs (Claude, Gemini, GPT, etc.).
- `vlm/` – adapters for open-source/local LVLMs (LLaVA, MiniGPT-4, InternVL, Ovis, Valley, etc.).
- `dataset/` – benchmark definitions, data preprocessing, and ground-truth metric functions.
- `inference.py`, `inference_mt.py`, `inference_video.py` – pipeline engines for single/multi-threaded image/video inference.
- `smp/` – shared multiprocessing utilities for parallel evaluation.
- `utils/` & `tools.py` – miscellaneous helpers (matching utilities, result transfer, logging).
- Scripts and notebooks under `scripts/` for custom data prep, mini-prototyping, and CI under `.github/`.

##### README for additional insights
- Describes the **evaluation pipeline**: data download → preprocessing → `generate_inner()` in VLM adapter → optional LLM-based answer extraction → metric scoring → result aggregation.
- Mentions a **config system** for flexible evaluation settings, and support for both **exact matching** and **LLM-based** metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes
    User(("User"))
    CLI["CLI / Entrypoint<br>(run.py)"]:::infra
    Config["Config System<br>(vlmeval/config.py)"]:::infra
    Dataset["Dataset Manager<br>(vlmeval/dataset/)"]:::data
    subgraph "Inference Engine"
        direction TB
        IE1["inference.py"]:::process
        IE2["inference_mt.py"]:::process
        IE3["inference_video.py"]:::process
    end
    API["API Wrappers<br>(vlmeval/api/)"]:::model
    Local["Local VLM Adapters<br>(vlmeval/vlm/)"]:::model
    Metrics["Metrics & Scoring<br>(vlmeval/dataset/utils/)"]:::metrics
    Result["Result Aggregator<br>(result_transfer.py)"]:::infra
    MP["Multiprocessing Utils<br>(vlmeval/smp/)"]:::infra
    Utils["General Utils<br>(vlmeval/utils/)"]:::infra
    Tools["tools.py"]:::infra

    subgraph "External Services"
        direction TB
        HF["HuggingFace Hub / ModelScope"]:::external
        RemoteAPI["Remote Model APIs"]:::external
        Dashboards["Remote Dashboards"]:::external
    end

    subgraph "Extension Points"
        direction TB
        AddBM["+ Add Benchmark"]:::infra
        AddMA["+ Add Model Adapter"]:::infra
    end

    %% Flow
    User -->|"runs"| CLI
    CLI -->|"loads"| Config
    Config -->|"orchestrates"| Dataset
    Dataset -->|"fetch & preprocess"| HF
    Dataset -->|"provides data"| IE1
    Dataset -->|"provides data"| IE2
    Dataset -->|"provides data"| IE3
    IE1 -->|"calls"| API
    IE2 -->|"calls"| API
    IE3 -->|"calls"| API
    IE1 -->|"calls"| Local
    IE2 -->|"calls"| Local
    IE3 -->|"calls"| Local
    API -->|"API call → JSON"| RemoteAPI
    Local -->|"model load"| HF
    IE1 -->|"predictions"| Metrics
    IE2 -->|"predictions"| Metrics
    IE3 -->|"predictions"| Metrics
    Metrics -->|"scores"| Result
    Result -->|"output JSON"| Dashboards

    %% Utilities support
    MP -.-> CLI
    MP -.-> Dataset
    MP -.-> IE1
    MP -.-> IE2
    MP -.-> IE3
    Utils -.-> Metrics
    Utils -.-> Result
    Tools -.-> CLI

    %% Extension arrows
    AddBM --> Dataset
    AddMA --> API
    AddMA --> Local

    %% Click events
    click CLI "https://github.com/open-compass/vlmevalkit/blob/main/run.py"
    click Config "https://github.com/open-compass/vlmevalkit/blob/main/vlmeval/config.py"
    click Dataset "https://github.com/open-compass/vlmevalkit/tree/main/vlmeval/dataset/"
    click Metrics "https://github.com/open-compass/vlmevalkit/tree/main/vlmeval/dataset/utils/"
    click API "https://github.com/open-compass/vlmevalkit/tree/main/vlmeval/api/"
    click Local "https://github.com/open-compass/vlmevalkit/tree/main/vlmeval/vlm/"
    click IE1 "https://github.com/open-compass/vlmevalkit/blob/main/vlmeval/inference.py"
    click IE2 "https://github.com/open-compass/vlmevalkit/blob/main/vlmeval/inference_mt.py"
    click IE3 "https://github.com/open-compass/vlmevalkit/blob/main/vlmeval/inference_video.py"
    click MP "https://github.com/open-compass/vlmevalkit/tree/main/vlmeval/smp/"
    click Utils "https://github.com/open-compass/vlmevalkit/tree/main/vlmeval/utils/"
    click Tools "https://github.com/open-compass/vlmevalkit/blob/main/vlmeval/tools.py"
    click Result "https://github.com/open-compass/vlmevalkit/blob/main/vlmeval/utils/result_transfer.py"

    %% Classes
    classDef infra fill:#dddddd,stroke:#888888
    classDef data fill:#bbdefb,stroke:#2196f3
    classDef model fill:#c8e6c9,stroke:#4caf50
    classDef process fill:#ffe0b2,stroke:#ff9800
    classDef metrics fill:#e1bee7,stroke:#9c27b0
    classDef external fill:#f5f5f5,stroke:#000000
```
