How stevejpapad/visual-incompatibility-transformer works
Based on our detailed analysis of the file tree and the README, this project is a research prototype for detecting visual incompatibility in fashion outfits using transformers and computer vision models. It is not a traditional full-stack application but rather a deep learning/experimental project that involves several stages: data preparation, feature extraction (using FLIP and external models like CLIP), and multi-task training (using VICTOR) to evaluate compatibility and detect mismatches in fashion items.
• Data & Dataset:
• “data” directory that stores the Polyvore dataset and supporting files (e.g., data/item_ids).
• Data pre-processing/preparation role of the script prepare_polyvore.py, which fetches and processes the Polyvore dataset to generate Polyvore-MISFITs.
• Data Loaders & Pre-processing:
• Feature Extraction & Pre-training Module (FLIP):
• FLIP component (FLIP.py) responsible for training computer vision models on fashion data, and for extracting visual features. Indicate that FLIP supports various image architectures such as resnet18, vit_base_patch32_224, etc.
• Dependency on external CLIP installation for text-image pre-training.
• Main Orchestration:
• Main.py as the orchestration script that triggers training runs. This script likely calls various functions in the FLIP, prepare_polyvore, and potentially VICTOR modules to execute experiments.
• Model Definitions & Architectures:
• Multi-Task Learning (VICTOR):
• VICTOR.py as the module that combines multi-modal inputs (visual and textual features) and runs the multi-task experiments, training the model that simultaneously predicts overall compatibility and detects specific mismatches.
• Dependencies & Configuration:
• An external configuration or dependency handler via requirements.txt and any instructions (e.g., setting data_path in main.py) that links to workflow management.
Relationships and Interactions
• Data flow should be depicted starting with raw input from the dataset in /data.
• Another component would show the data-loading pipeline (data_loaders.py) that feeds processed data to the training modules.
• The FLIP module (FLIP.py) is then used to pre-train or fine-tune computer vision models, extract both visual and textual features by interfacing with external models like CLIP.
• These features are then passed to the VICTOR module (VICTOR.py) for multi-task training and evaluation of outfit compatibility.
Architectural Patterns and Design Principles
• While not a traditional layered or MVC application, the project follows a modular design typical in research code where each Python file represents a movable block focused on a particular stage of the experimental pipeline.
• Separation of concerns:
• Data preparation and loading
• Model training (for both FLIP and VICTOR)
• Feature extraction and evaluation.
• A pattern of experimentation orchestration, where main.py acts as the script to run end-to-end experiments, driving both training and evaluation.
Important Technologies and Frameworks
• Python environment (>3.8) managed via venv/conda.
• Deep learning frameworks and libraries (likely PyTorch or TensorFlow, process mentioned in requirements.txt, though not explicitly stated).
• CLIP for contrastive language-image pre-training.
• Pre-trained vision models such as ResNet and transformer-based models.