# zasder3/train-clip-ft architecture

> Architecture diagram of the GitHub repository zasder3/train-clip-ft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zasder3/train-clip-ft
- Repository: https://github.com/zasder3/train-clip-ft

## Overview

##### Project type and purpose
- This is a machine-learning training pipeline (Python/PyTorch Lightning) focused on finetuning OpenAI’s CLIP models on custom image–text pairs.
- The README describes usage, dependencies, DataModule, model wrapper, and a training script (`train_finetune.py`).

##### File structure
Top-level files
- LICENSE, README.md: project metadata.
- train_finetune.py: entry point for launching training.
Directories
- data/
- text_image_dm.py: a Lightning DataModule that ingests image/text pairs from disk.
- models/
- configs/ (RN.yaml, ViT.yaml): model-specific hyperparameters and architecture choices.
- model.py: defines the core LightningModule subclass that implements forward, loss, optimizer.
- wrapper.py: loads & wraps a CLIP backbone (ResNet or ViT) into the LightningModule.
Other
- images/: for README illustrations.
- models/__pycache__ and data/__pycache__: compiled artifacts.

##### README for additional insights
- Dependencies: PyTorch Lightning, cosine-annealing scheduler.
- Usage: CLI flags for data folder, batch size, GPU selection.
- Extendibility: swap in a custom DataLoader by editing `trainer.fit(...)`.

a. Main components
1. Data Storage (local folder of image/text pairs)
2. DataModule (`data/text_image_dm.py`)
3. Configuration Store (`models/configs/*.yaml`)
4. Model Wrapper (`models/wrapper.py`)
5. LightningModule (`models/model.py`)
6. Training Script (`train_finetune.py`)
7. External Libraries & Services (PyTorch, PyTorch Lightning, cosine-annealing scheduler, CLIP API)
8. Compute Resources (CPU/GPU)
9. Output (trained checkpoint/model artifacts)
b. Relationships and interactions
- Storage → DataModule: DataModule reads images and captions.
- Config files → Model Wrapper & LightningModule: hyperparameters & architecture get injected at instantiation.
- train_finetune.py orchestrates:
- loads DataModule
- loads wrapper (which calls `clip.load(...)`)
- constructs LightningModule
- passes both to PyTorch Lightning Trainer
- Trainer ↔ GPU/CPU: executes training loop.
- Trainer → Output: writes checkpoints to disk.
c. Architectural patterns and principles
- Modular design: clear separation between data ingestion, model definition, and training orchestration.
- Configuration-driven: YAML files decouple hyperparameters and architecture choices.
- Extensible: DataModule and LightningModule interfaces allow easy swapping.
d. Key technologies and libraries
- PyTorch Lightning: training orchestration, GPU management, checkpointing.
- OpenAI CLIP library (`clip.load`)
- pytorch-cosine-annealing-with-warmup: learning rate scheduler.

##### Detail
- Module and its inputs/outputs.
- Where config parameters flow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DataStorage[(Data Storage)]:::data
        DataModule["DataModule"]:::data
    end

    %% Config Layer
    subgraph "Config Layer"
        ConfigStore["Config Store"]:::config
    end

    %% Model Layer
    subgraph "Model Layer"
        Wrapper["Model Wrapper"]:::model
        LightningModel["LightningModule"]:::model
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        TrainScript["train_finetune.py"]:::orchestration
        Trainer["PyTorch Lightning Trainer"]:::orchestration
    end

    %% Execution & External
    Compute["Compute (CPU/GPU)"]:::external
    ExternalLibs["External Libraries\n(PyTorch, CLIP, Scheduler)"]:::external

    %% Output
    Output["Output Checkpoint"]:::output

    %% Relationships
    DataStorage -->|reads| DataModule
    DataModule -->|feeds batches| Trainer
    ConfigStore -->|provides params| Wrapper
    ConfigStore -->|provides params| LightningModel
    TrainScript -->|instantiates| DataModule
    TrainScript -->|instantiates| Wrapper
    TrainScript -->|constructs| LightningModel
    TrainScript -->|calls| Trainer
    Wrapper -->|wraps| LightningModel
    Wrapper -->|uses clip.load| ExternalLibs
    LightningModel -->|uses scheduler| ExternalLibs
    Trainer -->|trains on| Compute
    Trainer -->|trains| LightningModel
    Trainer -->|saves| Output

    %% Click Events
    click DataStorage "https://github.com/zasder3/train-clip-ft/tree/main/data/"
    click DataModule "https://github.com/zasder3/train-clip-ft/blob/main/data/text_image_dm.py"
    click ConfigStore "https://github.com/zasder3/train-clip-ft/tree/main/models/configs/"
    click Wrapper "https://github.com/zasder3/train-clip-ft/blob/main/models/wrapper.py"
    click LightningModel "https://github.com/zasder3/train-clip-ft/blob/main/models/model.py"
    click TrainScript "https://github.com/zasder3/train-clip-ft/blob/main/train_finetune.py"

    %% Styles
    classDef data fill:#D6EAF8,stroke:#3498DB
    classDef config fill:#FCF3CF,stroke:#F1C40F
    classDef model fill:#D5F5E3,stroke:#27AE60
    classDef orchestration fill:#FDEDEC,stroke:#E74C3C
    classDef external fill:#F2F3F4,stroke:#7F8C8D
    classDef output fill:#EBDEF0,stroke:#8E44AD
```
