How ma-kjh/cma-oodd works
PROJECT TYPE & PURPOSE
• The project is an open-source academic implementation aimed at advancing OoD detection by fine-tuning large-scale vision-language models. It does so by enhancing cross-modal alignment between image and text embeddings.
• The repository integrates pre-trained CLIP components, employs negative labels (derived from WordNet) for OoD evaluation, and demonstrates both training and inference pipelines.
ANALYZING THE FILE STRUCTURE
• Top-level directories and files:
• “clip” folder: Contains the core CLIP model files (model.py, clip.py, simple_tokenizer.py, and a vocabulary file) for handling multi-modal embeddings.
• “Neglabel” folder: Stores negative label data and associated text files (grouped by grammatical/type categories) that are used to support the negmining method.
• “utils” folder: Contains helper modules for common functions, model loading, training utilities, metrics, etc.
• Scripts such as train.py, inference.py, and prompt.py that indicate the training and evaluation procedures.
• Additional resources like environment.yaml for dependency management and a README that explains dataset preparation and configuration.
CORE ARCHITECTURAL COMPONENTS
a. Data & Input Pipeline:
• DATA PREPARATION: Separate input directories for in-distribution (ImageNet-1k) datasets and OoD datasets.
• NEGATIVE LABEL DATA: Negative label files under “Neglabel/txtfiles” for enhanced evaluation.
b. Model Components:
• CLIP Backbone:
• Uses the provided “clip” modules to extract image embeddings and tokenize/process text.
• Incorporates modules for normalizing embeddings and computing similarities.
• Cross-Modal Alignment (CMA) Module:
• Implements the CMA loss that regularizes image and text embeddings. This is illustrated in train.py and within the provided code snippet.
• Uses a combination of loss functions (image_loss, text_loss) together with a regularization term built upon log-sum-exp functions over embeddings.
c. Training & Fine-tuning Pipeline:
• TRAINING SCRIPT (train.py):
• Handles multi-modal fine-tuning and the application of the CMA loss.
• Ingests in-distribution data and negative labels, performs normalization on embeddings, and adjusts scale factors.
• LOSS CALCULATION:
• Detailed in the README code snippet, showing the directional flow from input images/text to their embeddings, followed by computation of the final loss.
d. Inference & Evaluation Pipeline:
• INFERENCE SCRIPT (inference.py):
• Loads a pretrained checkpoint.
• Performs forward passes to compute image/text logits for OoD detection.
• EVALUATION:
• Uses metrics and algorithms defined under “utils/metrics.py.”
RELATIONSHIPS & DATA FLOW
• Incoming Data:
• In-distribution images (ImageNet-1k) and corresponding text descriptions (could be prompts).
• External OoD data from benchmark datasets.
• Flow of Data:
• Data ingestion → Preprocessing and tokenization (handled by CLIP’s simple_tokenizer.py and data loaders) → Passing data through the CLIP model (clip/model.py) for obtaining multi-modal embeddings.
• Embeddings are normalized, scaled, and then used in the loss computation block (CMA loss computation).
• The negative labels from “Neglabel” also feed into evaluation metrics or may be used in pairing/contrastive losses.
• Training vs. Inference:
• Training involves iterative adjustment of parameters using the training loop defined in train.py and associated utilities in “utils.”
• Inference routes through a simplified version of the model to evaluate OoD performance.
IMPORTANT ARCHITECTURAL PATTERNS/TECHNIQUES
• Multi-modal fine-tuning: Adjusts pretrained CLIP weights to better align image and text modalities.
• Cross-modal alignment: Emphasizes reducing the modality gap using a combined loss (including the log-sum-exp regularization term).
• Modularity: Clear segregation between core model functionality (in “clip”), data handling and negative labels (in “Neglabel”), and supporting utilities (in “utils”).