How miccunifi/searle works
Project Type and Purpose
• This is a research codebase (Python/PyTorch) implementing a zero-shot composed image retrieval method (“SEARLE”).
• It provides data-preprocessing, pseudo-token generation, two textual inversion strategies (OTI and learned network φ), and inference/evaluation on several datasets (FashionIQ, CIRR, CIRCO, ImageNet).
File Structure
Top-level directories and files
• data/ – raw and intermediate data, model checkpoints, submissions
• assets/ – paper poster & slides
• src/ – main Python modules (data_utils, datasets, gpt_phrases_generation, image_concepts_association, oti_inversion, train_phi, phi, encode_with_pseudo_tokens, validate, generate_test_submission, utils)
• hubconf.py – torch.hub integration for loading pre-trained SEARLE models
Patterns
• Linear pipeline: phrase generation → concept association → optimization-based inversion (OTI) → network training (φ) → inference/validation
• Clear separation of concerns by script
Extract Architecture Insights from README
• Dependencies: PyTorch, torchvision, transformers, openai CLIP, comet-ml, pandas
• External services: torch.hub (model download), OpenAI GPT-Neo for phrase generation, CLIP image/text encoders
• Multiple datasets with prescribed folder layouts
Main Components & Interactions
A. Data Storage
– Datasets: FashionIQ, CIRR, CIRCO, ImageNet (on disk under data/)
– Generated artifacts: GPT phrase pickle, associated concepts CSV, OTI pseudo-tokens, φ checkpoints, test submissions
B. Phrase Generation (src/gpt_phrases_generation.py)
– Input: concept list (Open Images V7)
– External: GPT-Neo model via HuggingFace transformers
– Output: data/GPT_phrases/<exp>/concept_to_phrases.pkl
C. Image–Concept Association (src/image_concepts_association.py)
– Input: images (ImageNet or CIRR/FashionIQ/CIRCO) + GPT phrases
– External: CLIP image encoder
– Output: CSV of top-K concepts per image
D. Optimization-Based Textual Inversion (OTI) (src/oti_inversion.py)
– Input: image features + associated concepts
– Process: gradient-based learning of pseudo-token embedding per image
– Output: data/oti_pseudo_tokens/…
E. Textual Inversion Network Training (src/train_phi.py & src/phi.py)
– Input: OTI pseudo-tokens + image features
– Model: small network φ that maps CLIP image features → pseudo-token embeddings
– Uses distillation & GPT loss
– Output: φ checkpoints under data/phi_models/…
F. Inference & Evaluation
– SEARLE hub model (hubconf.py) or φ model
– Scripts:
• src/validate.py – computes retrieval metrics on val set
• src/generate_test_submission.py – produces submission files for CIRR/CIRCO servers
• src/encode_with_pseudo_tokens.py – helper to integrate pseudo-tokens into CLIP text encoder
G. External Services & Tools
– torch.hub for model loading
– HuggingFace transformers (GPT-Neo)
– openai/CLIP repository
– comet-ml for optional logging
Architectural Patterns & Design Principles
• Pipeline / staged batch processing
• Separation of data preparation, model training, and inference/evaluation
• Modular CLI scripts for each stage, enabling reusability and easy swapping of components (e.g., different CLIP backbones, preprocess modes)
• Heavy reliance on on-disk artifacts to decouple stages and support resumability