How lions-epfl/vl-mia works
Project Type and Purpose
• This project is an academic/benchmarking codebase focused on membership inference attacks (MIAs) against large vision-language models (VLLMs).
• Its purpose is to provide a benchmark (VL-MIA) for evaluating when specific image or text data was part of a model’s training set, using methods like MaxRényi-K% and ModRényi metrics.
• It leverages several subcomponents that include multiple vision-language models (e.g., MiniGPT-4, llava, llama_adapter_v21) and supports both image-based runs and text-based runs.
File Structure
• Top-level directories include configuration files (.devcontainer, .dockerignore, etc.), and three primary directories with heavy development: “MiniGPT-4”, “llama_adapter_v21”, and “llava.” Each directory encapsulates a variant or support system for vision-language models.
• “MiniGPT-4” contains multiple sub-folders such as “common”, “configs”, “datasets”, “models”, “processors”, “runners”, and “tasks,” which indicate a modular design often following patterns like configuration-based model execution and multi-task support.
• “llama_adapter_v21” similarly contains files for finetuning, model adapter functionalities, tokenization, and evaluation, while “llava” includes components for evaluation, serving the model, training, and model building.
• Other utility scripts (eval.py, run_with_img.py, run_with_text.py) are directly present at the top level.
• The Docker and Conda files provide containerized or environment-specific deployment supporting reproducibility.