# ma-kjh/cma-oodd architecture

> Architecture diagram of the GitHub repository ma-kjh/cma-oodd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ma-kjh/cma-oodd
- Repository: https://github.com/ma-kjh/cma-oodd

## Overview

##### PROJECT TYPE & PURPOSE
- The project is an open-source academic implementation aimed at advancing OoD detection by fine-tuning large-scale vision-language models. It does so by enhancing cross-modal alignment between image and text embeddings.
- The repository integrates pre-trained CLIP components, employs negative labels (derived from WordNet) for OoD evaluation, and demonstrates both training and inference pipelines.

##### ANALYZING THE FILE STRUCTURE
- Top-level directories and files:
- “clip” folder: Contains the core CLIP model files (model.py, clip.py, simple_tokenizer.py, and a vocabulary file) for handling multi-modal embeddings.
- “Neglabel” folder: Stores negative label data and associated text files (grouped by grammatical/type categories) that are used to support the negmining method.
- “utils” folder: Contains helper modules for common functions, model loading, training utilities, metrics, etc.
- Scripts such as train.py, inference.py, and prompt.py that indicate the training and evaluation procedures.
- Additional resources like environment.yaml for dependency management and a README that explains dataset preparation and configuration.

##### CORE ARCHITECTURAL COMPONENTS
a. Data & Input Pipeline:
- DATA PREPARATION: Separate input directories for in-distribution (ImageNet-1k) datasets and OoD datasets.
- NEGATIVE LABEL DATA: Negative label files under “Neglabel/txtfiles” for enhanced evaluation.

b. Model Components:
- CLIP Backbone:
- Uses the provided “clip” modules to extract image embeddings and tokenize/process text.
- Incorporates modules for normalizing embeddings and computing similarities.
- Cross-Modal Alignment (CMA) Module:
- Implements the CMA loss that regularizes image and text embeddings. This is illustrated in train.py and within the provided code snippet.
- Uses a combination of loss functions (image_loss, text_loss) together with a regularization term built upon log-sum-exp functions over embeddings.

c. Training & Fine-tuning Pipeline:
- TRAINING SCRIPT (train.py):
- Handles multi-modal fine-tuning and the application of the CMA loss.
- Ingests in-distribution data and negative labels, performs normalization on embeddings, and adjusts scale factors.
- LOSS CALCULATION:
- Detailed in the README code snippet, showing the directional flow from input images/text to their embeddings, followed by computation of the final loss.

d. Inference & Evaluation Pipeline:
- INFERENCE SCRIPT (inference.py):
- Loads a pretrained checkpoint.
- Performs forward passes to compute image/text logits for OoD detection.
- EVALUATION:
- Uses metrics and algorithms defined under “utils/metrics.py.”

##### RELATIONSHIPS & DATA FLOW
- Incoming Data:
- In-distribution images (ImageNet-1k) and corresponding text descriptions (could be prompts).
- External OoD data from benchmark datasets.
- Flow of Data:
- Data ingestion → Preprocessing and tokenization (handled by CLIP’s simple_tokenizer.py and data loaders) → Passing data through the CLIP model (clip/model.py) for obtaining multi-modal embeddings.
- Embeddings are normalized, scaled, and then used in the loss computation block (CMA loss computation).
- The negative labels from “Neglabel” also feed into evaluation metrics or may be used in pairing/contrastive losses.
- Training vs. Inference:
- Training involves iterative adjustment of parameters using the training loop defined in train.py and associated utilities in “utils.”
- Inference routes through a simplified version of the model to evaluate OoD performance.

##### IMPORTANT ARCHITECTURAL PATTERNS/TECHNIQUES
- Multi-modal fine-tuning: Adjusts pretrained CLIP weights to better align image and text modalities.
- Cross-modal alignment: Emphasizes reducing the modality gap using a combined loss (including the log-sum-exp regularization term).
- Modularity: Clear segregation between core model functionality (in “clip”), data handling and negative labels (in “Neglabel”), and supporting utilities (in “utils”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    A["In-Distribution Data (ImageNet-1k)"]:::data
    B["Out-of-Distribution Data (OoD)"]:::data

    subgraph "Negative Label Data"
        C1["txtfiles"]:::data
        C2["neg_label_10000.npy"]:::data
    end

    %% Preprocessing Layer
    D["Tokenization & Preprocessing"]:::processing

    %% Core Model: CLIP Backbone Subgraph
    subgraph "CLIP Backbone"
        E1["clip.py"]:::core
        E2["model.py"]:::core
        E3["simple_tokenizer.py"]:::core
        E4["bpe_simple_vocab_16e6.txt.gz"]:::core
    end

    %% Cross-Modal Alignment Module
    F["Cross-Modal Alignment (CMA) Module"]:::core

    %% Training Pipeline & Loss Computation Subgraph
    subgraph "Training Pipeline & Loss Computation"
        G1["train.py"]:::training
        G2["train_utils.py"]:::training
    end

    %% Inference Pipeline
    H["Inference Pipeline (inference.py)"]:::inference

    %% Utility Modules Subgraph
    subgraph "Utility Modules"
        I1["common.py"]:::utils
        I2["load_model.py"]:::utils
        I3["loader.py"]:::utils
    end

    %% Evaluation Metrics Module
    J["Evaluation Metrics (metrics.py)"]:::utils

    %% Data Flow Connections
    A --> D
    D --> E1
    D --> E2
    D --> E3
    D --> E4

    E1 --> F
    E2 --> F
    E3 --> F
    E4 --> F

    F --> G1
    F --> G2

    C1 --> G1
    C2 --> G1

    E1 --> H
    B --> H

    G1 --> I1
    G2 --> I1
    H --> I2

    I1 --> J
    I2 --> J
    I3 --> J
    H --> J

    %% Click Events for CLIP Backbone components
    click E1 "https://github.com/ma-kjh/cma-oodd/blob/main/clip/clip.py"
    click E2 "https://github.com/ma-kjh/cma-oodd/blob/main/clip/model.py"
    click E3 "https://github.com/ma-kjh/cma-oodd/blob/main/clip/simple_tokenizer.py"
    click E4 "https://github.com/ma-kjh/cma-oodd/blob/main/clip/bpe_simple_vocab_16e6.txt.gz"

    %% Click Events for Negative Label Data components
    click C1 "https://github.com/ma-kjh/cma-oodd/tree/main/Neglabel/txtfiles"
    click C2 "https://github.com/ma-kjh/cma-oodd/blob/main/Neglabel/neg_label_10000.npy"

    %% Click Event for In-Distribution Data Handling
    click A "https://github.com/ma-kjh/cma-oodd/tree/main/CLIP_im1k_features"

    %% Click Events for Training Pipeline components
    click G1 "https://github.com/ma-kjh/cma-oodd/blob/main/train.py"
    click G2 "https://github.com/ma-kjh/cma-oodd/blob/main/utils/train_utils.py"

    %% Click Event for Inference Pipeline
    click H "https://github.com/ma-kjh/cma-oodd/blob/main/inference.py"

    %% Click Event for Evaluation Metrics
    click J "https://github.com/ma-kjh/cma-oodd/blob/main/utils/metrics.py"

    %% Click Events for Utility Modules
    click I1 "https://github.com/ma-kjh/cma-oodd/blob/main/utils/common.py"
    click I2 "https://github.com/ma-kjh/cma-oodd/blob/main/utils/load_model.py"
    click I3 "https://github.com/ma-kjh/cma-oodd/blob/main/utils/loader.py"

    %% Style Definitions
    classDef data fill:#FFF2CC,stroke:#333,stroke-width:2px;
    classDef processing fill:#CCE5FF,stroke:#333,stroke-width:2px;
    classDef core fill:#D5A6BD,stroke:#333,stroke-width:2px;
    classDef training fill:#B6D7A8,stroke:#333,stroke-width:2px;
    classDef inference fill:#F9CB9C,stroke:#333,stroke-width:2px;
    classDef utils fill:#C9DAF8,stroke:#333,stroke-width:2px;
```
