How fawazsammani/nlxgpt works
Project Type and Purpose
• This project is a research codebase for NLX-GPT—a model that generates natural language explanations for various vision and vision-language tasks (e.g., VQA-X, ACT-X, e-SNLI-VE, VCR). It supports training, predicting explanations, and even adversarial/retrieval evaluations.
• The README confirms that the system is built around fine-tuning and evaluation of language models (with a focus on natural language explanations) in combination with visual tasks.
Analyzing the File Structure
• Top-Level Scripts: Files like actX.py, esnlive.py, esnlive_concepts.py, vqaX.py, and vcr.py are the main entry points to run experiments for different tasks.
• Model Components: The “models” directory contains the core network modules (e.g., clip_vit.py, gpt.py, gpt_vcr.py) used to build the model architecture—indicating a reliance on both vision (CLIP/ViT) and language (GPT) models.
• Additional Functionalities:
– “explain_predict” folder: Contains scripts (ep_actX.py, ep_esnlive.py, ep_vqaX.py) and a README on generating predictions and explanations.
– “retrieval_attack” folder: Contains functionality for evaluation metrics, likely to test robustness or to perform retrieval-based attacks on the model.
– “utils” folder: Includes helper modules such as data_utils.py and eval_utils.py, as well as notebooks for preprocessing data and evaluation (e.g., for Coco Caption evaluation).
• Other Scripts: pretrain.py shows that there is functionality for pretraining, and supporting external evaluation or filtering (e.g., vcr_filter.py).
Main Components
a. Data Ingestion & Preprocessing:
– Inputs: Images and annotations from various datasets (as described in the README) including COCO, MPI, Flickr30k, and VCR.
– Utilities/Notebooks: Located in “utils” (e.g., nle_preprocess.ipynb, preprocess_for_cococaption_eval.ipynb) showing data cleaning and preparation steps.
b. Core Model Architecture:
– Model Modules: Components from “models” (CLIP-based vision module, GPT language model modules, and a specialized variant for VCR).
– Pretraining Layer: pretrain.py showcases the pretraining process (with potential use of pretrained captioning models).
c. Training and Finetuning Pipelines:
– Task-specific entry points: actX.py, esnlive.py (+ concepts variant), vqaX.py, and vcr.py each drive finetuning for the respective dataset task.
– Distributed Training: Integration with ‘accelerate’ for multi-GPU or distributed setups.
d. Prediction and Evaluation:
– Prediction Generation: The “explain_predict” folder handles generating and perhaps visualizing natural language explanations.
– Evaluation: Retrieval_attack and additional scripts (vcr_filter.py) handle evaluation and filtering, including using external tools (e.g., BERTScore, coco-caption).
e. External Interfaces and Dependencies:
– Gradio Demos: Mentioned as web-demo configurations for VQA-X and ACT-X.
– External Dependencies: PyTorch, CLIP, transformers, accelerate, and coco-caption for evaluation.
Relationships and Data Flow
• Data Flow: Start with raw images and annotations (downloadable as described in the README) → processed through preprocessing utilities → fed into the model during training (using pretrain.py and task-specific scripts) → generate predictions & explanations via explain_predict scripts → evaluate with retrieval_attack and filtering steps (vcr_filter.py, BERTScore, coco-caption).
• Components Interaction: The central NLX-GPT model (combining vision and language modules) is used by multiple pipelines. Training scripts call the model components; evaluation utilities leverage model predictions to gauge performance.