# miccunifi/searle architecture

> Architecture diagram of the GitHub repository miccunifi/searle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/miccunifi/searle
- Repository: https://github.com/miccunifi/searle

## Overview

##### Project Type and Purpose
- This is a research codebase (Python/PyTorch) implementing a zero-shot composed image retrieval method (“SEARLE”).
- It provides data-preprocessing, pseudo-token generation, two textual inversion strategies (OTI and learned network φ), and inference/evaluation on several datasets (FashionIQ, CIRR, CIRCO, ImageNet).

##### File Structure
Top-level directories and files
- data/ – raw and intermediate data, model checkpoints, submissions
- assets/ – paper poster & slides
- src/ – main Python modules (data_utils, datasets, gpt_phrases_generation, image_concepts_association, oti_inversion, train_phi, phi, encode_with_pseudo_tokens, validate, generate_test_submission, utils)
- hubconf.py – torch.hub integration for loading pre-trained SEARLE models
Patterns
- Linear pipeline: phrase generation → concept association → optimization-based inversion (OTI) → network training (φ) → inference/validation
- Clear separation of concerns by script

##### Extract Architecture Insights from README
- Dependencies: PyTorch, torchvision, transformers, openai CLIP, comet-ml, pandas
- External services: torch.hub (model download), OpenAI GPT-Neo for phrase generation, CLIP image/text encoders
- Multiple datasets with prescribed folder layouts

##### Main Components & Interactions
A. Data Storage
– Datasets: FashionIQ, CIRR, CIRCO, ImageNet (on disk under data/)
– Generated artifacts: GPT phrase pickle, associated concepts CSV, OTI pseudo-tokens, φ checkpoints, test submissions
B. Phrase Generation (src/gpt_phrases_generation.py)
– Input: concept list (Open Images V7)
– External: GPT-Neo model via HuggingFace transformers
– Output: data/GPT_phrases/<exp>/concept_to_phrases.pkl
C. Image–Concept Association (src/image_concepts_association.py)
– Input: images (ImageNet or CIRR/FashionIQ/CIRCO) + GPT phrases
– External: CLIP image encoder
– Output: CSV of top-K concepts per image
D. Optimization-Based Textual Inversion (OTI) (src/oti_inversion.py)
– Input: image features + associated concepts
– Process: gradient-based learning of pseudo-token embedding per image
– Output: data/oti_pseudo_tokens/…
E. Textual Inversion Network Training (src/train_phi.py & src/phi.py)
– Input: OTI pseudo-tokens + image features
– Model: small network φ that maps CLIP image features → pseudo-token embeddings
– Uses distillation & GPT loss
– Output: φ checkpoints under data/phi_models/…
F. Inference & Evaluation
– SEARLE hub model (hubconf.py) or φ model
– Scripts:
- src/validate.py – computes retrieval metrics on val set
- src/generate_test_submission.py – produces submission files for CIRR/CIRCO servers
- src/encode_with_pseudo_tokens.py – helper to integrate pseudo-tokens into CLIP text encoder
G. External Services & Tools
– torch.hub for model loading
– HuggingFace transformers (GPT-Neo)
– openai/CLIP repository
– comet-ml for optional logging

##### Architectural Patterns & Design Principles
- Pipeline / staged batch processing
- Separation of data preparation, model training, and inference/evaluation
- Modular CLI scripts for each stage, enabling reusability and easy swapping of components (e.g., different CLIP backbones, preprocess modes)
- Heavy reliance on on-disk artifacts to decouple stages and support resumability

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Storage
    subgraph "Data Storage"
        DS[(data/)]:::db
    end

    %% Compute Modules
    GP["gpt_phrases_generation.py<br/>(GPT-Neo)"]:::compute
    IA["image_concepts_association.py<br/>(CLIP)"]:::compute
    OTI["oti_inversion.py<br/>(OTI)"]:::compute
    subgraph "Textual Inversion Network"
        phiDef["phi.py<br/>(φ Definition)"]:::compute
        phiTrain["train_phi.py<br/>(φ Training)"]:::compute
    end
    Encode["encode_with_pseudo_tokens.py"]:::compute
    Inference["hubconf.py<br/>(SEARLE Model + CLIP)"]:::compute
    Validate["validate.py<br/>(Validation)"]:::compute
    Submit["generate_test_submission.py<br/>(Submission)"]:::compute

    %% External Services
    GPTService["HuggingFace GPT-Neo"]:::external
    CLIPService["OpenAI CLIP"]:::external
    TorchHub["torch.hub"]:::external

    %% Data Flow
    DS -->|images| IA
    DS -->|image features| OTI
    DS -->|checkpoints & submissions| Inference

    GPTService --> GP
    GP -->|concept_to_phrases.pkl| IA
    GP -->|concept_to_phrases.pkl| OTI

    CLIPService --> IA
    IA -->|CSV of top-K concepts| OTI

    OTI -->|OTI pseudo-tokens| phiTrain
    phiDef --> phiTrain

    phiTrain -->|φ checkpoint| Inference
    Encode --> Inference

    TorchHub --> Inference
    CLIPService --> Inference

    Inference -->|retrieval metrics| Validate
    Inference -->|submission JSON| Submit

    %% Click Events
    click DS "https://github.com/miccunifi/searle/tree/main/data/"
    click GP "https://github.com/miccunifi/searle/blob/main/src/gpt_phrases_generation.py"
    click IA "https://github.com/miccunifi/searle/blob/main/src/image_concepts_association.py"
    click OTI "https://github.com/miccunifi/searle/blob/main/src/oti_inversion.py"
    click phiDef "https://github.com/miccunifi/searle/blob/main/src/phi.py"
    click phiTrain "https://github.com/miccunifi/searle/blob/main/src/train_phi.py"
    click Encode "https://github.com/miccunifi/searle/blob/main/src/encode_with_pseudo_tokens.py"
    click Inference "https://github.com/miccunifi/searle/blob/main/hubconf.py"
    click Validate "https://github.com/miccunifi/searle/blob/main/src/validate.py"
    click Submit "https://github.com/miccunifi/searle/blob/main/src/generate_test_submission.py"

    %% Styles
    classDef db fill:#BBDEFB,stroke:#333,stroke-width:1px;
    classDef compute fill:#C8E6C9,stroke:#333,stroke-width:1px;
    classDef external fill:#FFCC80,stroke:#333,stroke-width:1px;
```
