# lions-epfl/vl-mia architecture

> Architecture diagram of the GitHub repository lions-epfl/vl-mia, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lions-epfl/vl-mia
- Repository: https://github.com/lions-epfl/vl-mia

## Overview

##### Project Type and Purpose
- This project is an academic/benchmarking codebase focused on membership inference attacks (MIAs) against large vision-language models (VLLMs).
- Its purpose is to provide a benchmark (VL-MIA) for evaluating when specific image or text data was part of a model’s training set, using methods like MaxRényi-K% and ModRényi metrics.
- It leverages several subcomponents that include multiple vision-language models (e.g., MiniGPT-4, llava, llama_adapter_v21) and supports both image-based runs and text-based runs.

##### File Structure
- Top-level directories include configuration files (.devcontainer, .dockerignore, etc.), and three primary directories with heavy development: “MiniGPT-4”, “llama_adapter_v21”, and “llava.” Each directory encapsulates a variant or support system for vision-language models.
- “MiniGPT-4” contains multiple sub-folders such as “common”, “configs”, “datasets”, “models”, “processors”, “runners”, and “tasks,” which indicate a modular design often following patterns like configuration-based model execution and multi-task support.
- “llama_adapter_v21” similarly contains files for finetuning, model adapter functionalities, tokenization, and evaluation, while “llava” includes components for evaluation, serving the model, training, and model building.
- Other utility scripts (eval.py, run_with_img.py, run_with_text.py) are directly present at the top level.
- The Docker and Conda files provide containerized or environment-specific deployment supporting reproducibility.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Flow
    HF["HuggingFace Datasets"]:::external
    DL["Data Loader: VL-MIA Datasets"]:::datasource
    P["Preprocessing"]:::processing
    HF -->|"ingests"| DL
    DL -->|"loads"| P

    %% Model Components Subgraph
    subgraph "Model Components"
        MGP["MiniGPT-4 (Model)"]:::model
        LA["Llama Adapter (Model)"]:::model
        LV["Llava (Model)"]:::model
    end
    P -->|"preprocess"| MGP
    P -->|"preprocess"| LA
    P -->|"preprocess"| LV

    %% Attack & Evaluation Subgraph
    subgraph "Attack & Evaluation"
        MIA["MI Attack Evaluator"]:::attack
        EP["Evaluation Pipeline"]:::evaluation
        UL["Utility & Logging"]:::logging
    end
    MGP -->|"inference"| MIA
    LA -->|"inference"| MIA
    LV -->|"inference"| MIA

    MIA -->|"evaluate"| EP
    MIA -->|"log"| UL

    %% Configurations (applied across layers)
    Conf["Configurations (YAML)"]:::processing
    Conf ---|"applies_to"| DL
    Conf ---|"applies_to"| MGP
    Conf ---|"applies_to"| EP

    %% Environment & Containerization Subgraph
    subgraph "Environment Layer"
        ENV["Environment & Containerization"]:::env
    end
    ENV ---|"supports"| DL
    ENV ---|"supports"| MGP
    ENV ---|"supports"| LA
    ENV ---|"supports"| LV
    ENV ---|"supports"| EP
    ENV ---|"supports"| UL

    %% Click Events
    click DL "https://github.com/lions-epfl/vl-mia/tree/main/MiniGPT-4/minigpt4/datasets"
    click MGP "https://github.com/lions-epfl/vl-mia/tree/main/MiniGPT-4/minigpt4/models"
    click LA "https://github.com/lions-epfl/vl-mia/tree/main/llama_adapter_v21/llama"
    click LV "https://github.com/lions-epfl/vl-mia/tree/main/llava/model"
    click MIA "https://github.com/lions-epfl/vl-mia/blob/main/eval.py"
    click EP "https://github.com/lions-epfl/vl-mia/tree/main/MiniGPT-4/eval_configs"
    click ENV "https://github.com/lions-epfl/vl-mia/blob/main/.devcontainer"
    click UL "https://github.com/lions-epfl/vl-mia/blob/main/MiniGPT-4/minigpt4/common/logger.py"

    %% Styles
    classDef external fill:#F0B27A,stroke:#D35400,stroke-width:2px,stroke-dasharray: 5,5
    classDef datasource fill:#AED6F1,stroke:#2471A3,stroke-width:2px
    classDef processing fill:#F9E79F,stroke:#F1C40F,stroke-width:2px
    classDef model fill:#A9DFBF,stroke:#239B56,stroke-width:2px
    classDef attack fill:#FAD7A0,stroke:#D35400,stroke-width:2px
    classDef evaluation fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px
    classDef env fill:#D6EAF8,stroke:#3498DB,stroke-width:2px
    classDef logging fill:#F5CBA7,stroke:#CA6F1E,stroke-width:2px
```
