# aaanhho/vl-mia architecture

> Architecture diagram of the GitHub repository aaanhho/vl-mia, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaanhho/vl-mia
- Repository: https://github.com/aaanhho/vl-mia

## Overview

##### And Isolate the Major Components
- Entry Points & Experiment Drivers: Highlight separate CLI entry scripts—run_with_img.py and run_with_text.py—which launch the respective MIA experiments.
- Core MIA Engine: Detail the modules responsible for calculating the membership inference metrics (e.g., the target-free metric MaxRényi-K% and the modified target-based ModRényi). Although these metrics might be computed through modules in the “common” directories or within evaluation scripts, denote a central processing block that orchestrates the analysis.
- MiniGPT-4: Its internal structure (configs, datasets, models, processors) indicates a modular design, with components handling image-text processing, conversation handling, and evaluation support.
- llama_adapter_v21 and llava: These directories include modules for training, evaluation, adaptation (e.g., for finetuning or inference), and service components (including the serving and experimentation utilities).
- Utility Modules: Include supporting libraries (e.g., common utilities, logging, configuration management, and helper scripts, as seen in directories like “common”, “datasets”, “processors”, and “tasks”). These facilitate everything from configuration loading to experiment tracking.
- Environment & Deployment: Briefly indicate the development environment setup (the presence of Dockerfiles, Conda environment files, and container configuration) which shows that containerization and reproducible environments are key parts of the setup.
- External Libraries & Data Flow: Clearly link external dependencies (like the Hugging Face datasets API and possibly deep learning libraries such as PyTorch) that interact with these internal modules.

##### Relationships and Data Flow
- That after inference, the results are passed into the membership inference evaluation module, where the metrics (MaxRényi-K% and ModRényi) are computed.
- Demonstrate that logging, evaluation scripts, and possibly configuration settings influence and monitor this data flow.

##### Architectural Patterns and Design Choices
- Modular Design: Highlight that the system is split into clear modules (datasets, models, tasks, evaluators). Each module is responsible for a specific piece of functionality.
- Extensibility: Indicate how the design supports adding new datasets or models (as seen by the dynamically defined configuration files under “configs/datasets” and “configs/models”).
- Experiment-Driven: Reinforce that the two main experiment drivers (run_with_img.py and run_with_text.py) orchestrate the end-to-end pipeline of dataset intake, model inference, and MIA evaluation.

##### Incorporate Relevant Technologies and Frameworks
- Python as the main programming language.
- Use of Conda and Docker for environment management.
- Hugging Face's datasets library for dataset handling.
- Likely use of deep learning frameworks such as PyTorch underlying the model implementations (even if not explicitly mentioned in the README, the file structure and typical research implementations suggest this).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    DS["VL-MIA Datasets (Hugging Face)"]:::external

    %% Entry Points & Experiment Drivers Subgraph
    subgraph "Experiment Drivers"
        TE_img["Top-level run_with_img.py"]:::entry
        TE_text["Top-level run_with_text.py"]:::entry
        MG_img["MiniGPT-4 run_with_img.py"]:::entry
        MG_text["MiniGPT-4 run_with_text.py"]:::entry
        LA_img["llama_adapter_v21 run_with_img.py"]:::entry
        LA_text["llama_adapter_v21 run_with_text.py"]:::entry
    end

    %% Data Management & Dataset Loading Subgraph
    subgraph "Data Management"
        DM1["Dataset Builders (MiniGPT-4/minigpt4/datasets)"]:::data
        DM2["Dataset Configs (MiniGPT-4/minigpt4/configs/datasets)"]:::data
    end

    %% Model Integration and Inference Subgraph
    subgraph "Model Integration"
        MI1["MiniGPT-4 Models"]:::model
        MI2["llama_adapter_v21 Models"]:::model
        MI3["llava Models & Inference"]:::model
    end

    %% Core MIA Engine Subgraph
    subgraph "MIA Evaluation"
        ME1["MIA Utils (MiniGPT-4/minigpt4/common/eval_utils.py)"]:::evaluation
        ME2["MIA Top-level eval (eval.py)"]:::evaluation
        ME3["llava MIA Eval (llava/eval/run_llava.py)"]:::evaluation
    end

    %% Utility Modules Subgraph
    subgraph "Utilities"
        U1["Common Utils (MiniGPT-4/minigpt4/common)"]:::utility
        U2["Task Runners (MiniGPT-4/minigpt4/tasks)"]:::utility
        U3["llama_adapter_v21 Util (llama_adapter_v21/util)"]:::utility
        U4["llava Utils (llava/utils.py)"]:::utility
        U5["llava MM Utils (llava/mm_utils.py)"]:::utility
    end

    %% Environment & Deployment Node
    ENV["Environment & Deployment (.devcontainer, environment.yml, Docker/Conda)"]:::environment

    %% Data Flow Connections
    DS -->|"feeds"| DM1
    DS -->|"feeds"| DM2

    TE_img -->|"triggers"| DM1
    TE_img -->|"triggers"| DM2
    TE_text -->|"triggers"| DM1
    TE_text -->|"triggers"| DM2
    MG_img -->|"triggers"| DM1
    MG_img -->|"triggers"| DM2
    MG_text -->|"triggers"| DM1
    MG_text -->|"triggers"| DM2
    LA_img -->|"triggers"| DM1
    LA_img -->|"triggers"| DM2
    LA_text -->|"triggers"| DM1
    LA_text -->|"triggers"| DM2

    DM1 -->|"data flows"| MI1
    DM2 -->|"configures"| MI1
    DM1 -->|"data flows"| MI2
    DM2 -->|"configures"| MI2
    DM1 -->|"data flows"| MI3
    DM2 -->|"configures"| MI3

    MI1 -->|"results to"| ME1
    MI2 -->|"results to"| ME1
    MI3 -->|"results to"| ME3
    MI1 -->|"results to"| ME2
    MI2 -->|"results to"| ME2
    MI3 -->|"results to"| ME2

    %% Utilities support flow (dotted lines)
    U1 ---|"supports"| DM1
    U1 ---|"supports"| MI1
    U1 ---|"supports"| ME1
    U2 ---|"triggers tasks for"| DM1
    U2 ---|"orchestrates"| MI1
    U3 ---|"assists"| MI2
    U4 ---|"assists"| MI3
    U5 ---|"assists"| MI3

    %% Environment influences (dotted lines)
    ENV ---|"configures"| TE_img
    ENV ---|"configures"| TE_text
    ENV ---|"configures"| DM1
    ENV ---|"configures"| MI1
    ENV ---|"configures"| ME1

    %% Styles
    classDef external fill:#ffe6b3,stroke:#cc6600,stroke-width:2px;
    classDef entry fill:#cce6ff,stroke:#337ab7,stroke-width:2px;
    classDef data fill:#d5e8d4,stroke:#82b366,stroke-width:2px;
    classDef model fill:#f8cecc,stroke:#b85450,stroke-width:2px;
    classDef evaluation fill:#fff2cc,stroke:#d6b656,stroke-width:2px;
    classDef utility fill:#e1d5e7,stroke:#9673a6,stroke-width:2px;
    classDef environment fill:#d9edf7,stroke:#31708f,stroke-width:2px;

    %% Click Events
    click TE_img "https://github.com/aaanhho/vl-mia/blob/main/run_with_img.py"
    click TE_text "https://github.com/aaanhho/vl-mia/blob/main/run_with_text.py"
    click MG_img "https://github.com/aaanhho/vl-mia/blob/main/MiniGPT-4/run_with_img.py"
    click MG_text "https://github.com/aaanhho/vl-mia/blob/main/MiniGPT-4/run_with_text.py"
    click LA_img "https://github.com/aaanhho/vl-mia/blob/main/llama_adapter_v21/run_with_img.py"
    click LA_text "https://github.com/aaanhho/vl-mia/blob/main/llama_adapter_v21/run_with_text.py"

    click DM1 "https://github.com/aaanhho/vl-mia/tree/main/MiniGPT-4/minigpt4/datasets"
    click DM2 "https://github.com/aaanhho/vl-mia/tree/main/MiniGPT-4/minigpt4/configs/datasets"

    click ME1 "https://github.com/aaanhho/vl-mia/blob/main/MiniGPT-4/minigpt4/common/eval_utils.py"
    click ME2 "https://github.com/aaanhho/vl-mia/blob/main/eval.py"
    click ME3 "https://github.com/aaanhho/vl-mia/blob/main/llava/eval/run_llava.py"

    click MI1 "https://github.com/aaanhho/vl-mia/tree/main/MiniGPT-4"
    click MI2 "https://github.com/aaanhho/vl-mia/tree/main/llama_adapter_v21"
    click MI3 "https://github.com/aaanhho/vl-mia/tree/main/llava"

    click U1 "https://github.com/aaanhho/vl-mia/tree/main/MiniGPT-4/minigpt4/common"
    click U2 "https://github.com/aaanhho/vl-mia/tree/main/MiniGPT-4/minigpt4/tasks"
    click U3 "https://github.com/aaanhho/vl-mia/tree/main/llama_adapter_v21/util"
    click U4 "https://github.com/aaanhho/vl-mia/blob/main/llava/utils.py"
    click U5 "https://github.com/aaanhho/vl-mia/blob/main/llava/mm_utils.py"

    click ENV "https://github.com/aaanhho/vl-mia/blob/main/.devcontainer"
    click ENV "https://github.com/aaanhho/vl-mia/blob/main/MiniGPT-4/environment.yml"
```
