# stevejpapad/visual-incompatibility-transformer architecture

> Architecture diagram of the GitHub repository stevejpapad/visual-incompatibility-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stevejpapad/visual-incompatibility-transformer
- Repository: https://github.com/stevejpapad/visual-incompatibility-transformer

## Overview

Based on our detailed analysis of the file tree and the README, this project is a research prototype for detecting visual incompatibility in fashion outfits using transformers and computer vision models. It is not a traditional full-stack application but rather a deep learning/experimental project that involves several stages: data preparation, feature extraction (using FLIP and external models like CLIP), and multi-task training (using VICTOR) to evaluate compatibility and detect mismatches in fashion items.

- Data & Dataset:
- “data” directory that stores the Polyvore dataset and supporting files (e.g., data/item_ids).
- Data pre-processing/preparation role of the script prepare_polyvore.py, which fetches and processes the Polyvore dataset to generate Polyvore-MISFITs.

- Data Loaders & Pre-processing:

- Feature Extraction & Pre-training Module (FLIP):
- FLIP component (FLIP.py) responsible for training computer vision models on fashion data, and for extracting visual features. Indicate that FLIP supports various image architectures such as resnet18, vit_base_patch32_224, etc.
- Dependency on external CLIP installation for text-image pre-training.

- Main Orchestration:
- Main.py as the orchestration script that triggers training runs. This script likely calls various functions in the FLIP, prepare_polyvore, and potentially VICTOR modules to execute experiments.

- Model Definitions & Architectures:

- Multi-Task Learning (VICTOR):
- VICTOR.py as the module that combines multi-modal inputs (visual and textual features) and runs the multi-task experiments, training the model that simultaneously predicts overall compatibility and detects specific mismatches.

- Dependencies & Configuration:
- An external configuration or dependency handler via requirements.txt and any instructions (e.g., setting data_path in main.py) that links to workflow management.

##### Relationships and Interactions
- Data flow should be depicted starting with raw input from the dataset in /data.
- Another component would show the data-loading pipeline (data_loaders.py) that feeds processed data to the training modules.
- The FLIP module (FLIP.py) is then used to pre-train or fine-tune computer vision models, extract both visual and textual features by interfacing with external models like CLIP.
- These features are then passed to the VICTOR module (VICTOR.py) for multi-task training and evaluation of outfit compatibility.

##### Architectural Patterns and Design Principles
- While not a traditional layered or MVC application, the project follows a modular design typical in research code where each Python file represents a movable block focused on a particular stage of the experimental pipeline.
- Separation of concerns:
- Data preparation and loading
- Model training (for both FLIP and VICTOR)
- Feature extraction and evaluation.
- A pattern of experimentation orchestration, where main.py acts as the script to run end-to-end experiments, driving both training and evaluation.

##### Important Technologies and Frameworks
- Python environment (>3.8) managed via venv/conda.
- Deep learning frameworks and libraries (likely PyTorch or TensorFlow, process mentioned in requirements.txt, though not explicitly stated).
- CLIP for contrastive language-image pre-training.
- Pre-trained vision models such as ResNet and transformer-based models.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Preparation Stage
    subgraph "Data Preparation"
        DP_Data["data (Polyvore dataset)"]:::dataset
        Prepare["prepare_polyvore.py (Preprocessing)"]:::preprocess
        DP_Data --> Prepare
    end

    %% Data Loading Stage
    subgraph "Data Loading"
        DataLoader["data_loaders.py (Data Loader)"]:::loader
    end

    %% Feature Extraction & Pre-training Stage
    subgraph "Feature Extraction & Pre-training"
        FLIPNode["FLIP.py (Feature Extraction/FLIP Model Training)"]:::feature
        CLIP["External CLIP Library"]:::external
        FLIPNode --> CLIP
    end

    %% Model Definitions Stage
    subgraph "Model Definitions"
        ModelDef["models.py (Model Definitions)"]:::model
    end

    %% Multi-Task Learning Stage
    subgraph "Multi-Task Learning"
        VICTORNode["VICTOR.py (Multi-Task Learning/VICTOR)"]:::multiTask
    end

    %% Experiment Orchestration Stage
    subgraph "Experiment Orchestration"
        MainNode["main.py (Experiment Orchestration)"]:::orchestration
    end

    %% Dependencies & Configuration
    subgraph "Dependencies & Configuration"
        Config["requirements.txt (Dependencies & Configuration)"]:::config
    end

    %% Vertical Pipeline Flow
    Prepare --> DataLoader
    DataLoader --> FLIPNode
    FLIPNode --> VICTORNode
    ModelDef --> VICTORNode

    %% Orchestration triggering connections (side interactions)
    MainNode -->|"triggers"| Prepare
    MainNode -->|"triggers"| DataLoader
    MainNode -->|"triggers"| FLIPNode
    MainNode -->|"triggers"| ModelDef
    MainNode -->|"triggers"| VICTORNode
    Config --> MainNode

    %% Click Events
    click DP_Data "https://github.com/stevejpapad/visual-incompatibility-transformer/tree/main/data"
    click Prepare "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/prepare_polyvore.py"
    click DataLoader "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/data_loaders.py"
    click FLIPNode "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/FLIP.py"
    click ModelDef "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/models.py"
    click VICTORNode "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/VICTOR.py"
    click MainNode "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/main.py"
    click Config "https://github.com/stevejpapad/visual-incompatibility-transformer/blob/main/requirements.txt"

    %% Styles
    classDef dataset fill:#f9c,stroke:#333,stroke-width:2px;
    classDef preprocess fill:#bbf,stroke:#333,stroke-width:2px;
    classDef loader fill:#afa,stroke:#333,stroke-width:2px;
    classDef feature fill:#fdd,stroke:#333,stroke-width:2px;
    classDef model fill:#dfd,stroke:#333,stroke-width:2px;
    classDef multiTask fill:#ffd,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#cff,stroke:#333,stroke-width:2px;
    classDef config fill:#fcf,stroke:#333,stroke-width:2px;
    classDef external fill:#eee,stroke:#333,stroke-dasharray: 5 5,stroke-width:2px;
```
