# arrowluo/clip4clip architecture

> Architecture diagram of the GitHub repository arrowluo/clip4clip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arrowluo/clip4clip
- Repository: https://github.com/arrowluo/clip4clip

## Overview

CLIP4Clip trains and evaluates video–text retrieval models across five benchmark datasets. The main workflow is to select a dataset loader, batch caption/video examples, encode them with CLIP4Clip, optimize retrieval training, and evaluate retrieval metrics. The model supports parameter-free, sequential, and tight similarity approaches, with configurable frame sampling and 2D/3D patches. Edges follow the excerpts, SOURCE INDEX references, or README; unsampled implementation details are not inferred.

Main entry points: [`main_task_retrieval.py`](https://github.com/arrowluo/clip4clip/blob/master/main_task_retrieval.py) (Retrieval runner), [`dataloaders/data_dataloaders.py`](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/data_dataloaders.py) (Loader factory), [`dataloaders/rawvideo_util.py`](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/rawvideo_util.py) (Frame extraction), [`modules/modeling.py`](https://github.com/arrowluo/clip4clip/blob/master/modules/modeling.py) (CLIP4Clip model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Retrieval workflow"]
  node_runner["Retrieval runner"]
  node_batching["Loader factory"]
  node_checkpoint["Model checkpoint"]
end

subgraph group_datasets["Dataset ingestion"]
  node_msrvtt["MSR-VTT data"]
  node_msvd["MSVD data"]
  node_lsmdc["LSMDC data"]
  node_activitynet["ActivityNet data"]
  node_didemo["DiDeMo data"]
  node_frames["Frame extraction<br/>[rawvideo_util.py]"]
end

subgraph group_model["Model and similarity"]
  node_tokenizer["CLIP tokenizer"]
  node_retrieval_model["CLIP4Clip model<br/>[modeling.py]"]
  node_clip_encoder["CLIP encoder<br/>[module_clip.py]"]
  node_cross_encoder["Cross transformer<br/>[module_cross.py]"]
  node_model_primitives["Model primitives<br/>[until_module.py]"]
  node_config["Model configuration<br/>[until_config.py]"]
  node_cache["Model file cache<br/>[file_utils.py]"]
end

subgraph group_support["Training support"]
  node_optimizer["Optimizer schedule<br/>[optimization.py]"]
  node_metrics["Retrieval metrics<br/>[metrics.py]"]
  node_utilities["Runtime utilities<br/>[util.py]"]
end

node_user(("Researcher"))

node_user -->|"configures and runs"| node_runner
node_runner -->|"requests batches"| node_batching
node_batching -->|"constructs loader"| node_msrvtt
node_batching -->|"constructs loader"| node_msvd
node_batching -->|"constructs loader"| node_lsmdc
node_batching -->|"constructs loader"| node_activitynet
node_batching -->|"constructs loader"| node_didemo
node_msrvtt -->|"extracts frames"| node_frames
node_msvd -->|"extracts frames"| node_frames
node_lsmdc -->|"extracts frames"| node_frames
node_activitynet -->|"extracts frames"| node_frames
node_didemo -->|"extracts frames"| node_frames
node_runner -->|"creates tokenizer"| node_tokenizer
node_runner -->|"initializes and trains"| node_retrieval_model
node_retrieval_model -->|"uses encoder"| node_clip_encoder
node_retrieval_model -->|"uses transformer"| node_cross_encoder
node_cross_encoder -->|"uses layers"| node_model_primitives
node_retrieval_model -->|"loads configuration"| node_config
node_config -->|"resolves archives"| node_cache
node_cross_encoder -->|"resolves pretrained files"| node_cache
node_runner -->|"updates parameters"| node_optimizer
node_runner -->|"evaluates retrieval"| node_metrics
node_runner -->|"uses runtime helpers"| node_utilities
node_runner -->|"saves or resumes"| node_checkpoint

click node_runner "https://github.com/arrowluo/clip4clip/blob/master/main_task_retrieval.py"
click node_batching "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/data_dataloaders.py"
click node_msrvtt "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_msrvtt_retrieval.py"
click node_msvd "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_msvd_retrieval.py"
click node_lsmdc "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_lsmdc_retrieval.py"
click node_activitynet "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_activitynet_retrieval.py"
click node_didemo "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_didemo_retrieval.py"
click node_frames "https://github.com/arrowluo/clip4clip/blob/master/dataloaders/rawvideo_util.py"
click node_tokenizer "https://github.com/arrowluo/clip4clip/blob/master/modules/tokenization_clip.py"
click node_retrieval_model "https://github.com/arrowluo/clip4clip/blob/master/modules/modeling.py"
click node_clip_encoder "https://github.com/arrowluo/clip4clip/blob/master/modules/module_clip.py"
click node_cross_encoder "https://github.com/arrowluo/clip4clip/blob/master/modules/module_cross.py"
click node_model_primitives "https://github.com/arrowluo/clip4clip/blob/master/modules/until_module.py"
click node_config "https://github.com/arrowluo/clip4clip/blob/master/modules/until_config.py"
click node_cache "https://github.com/arrowluo/clip4clip/blob/master/modules/file_utils.py"
click node_optimizer "https://github.com/arrowluo/clip4clip/blob/master/modules/optimization.py"
click node_metrics "https://github.com/arrowluo/clip4clip/blob/master/metrics.py"
click node_utilities "https://github.com/arrowluo/clip4clip/blob/master/util.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_batching,node_checkpoint toneBlue
class node_msrvtt,node_msvd,node_lsmdc,node_activitynet,node_didemo,node_frames toneAmber
class node_tokenizer,node_retrieval_model,node_clip_encoder,node_cross_encoder,node_model_primitives,node_config,node_cache toneMint
class node_optimizer,node_metrics,node_utilities toneRose
class node_user toneIndigo
```

## Components

### Retrieval workflow

- [Retrieval runner](https://github.com/arrowluo/clip4clip/blob/master/main_task_retrieval.py): `main_task_retrieval.py`
- [Loader factory](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/data_dataloaders.py): `dataloaders/data_dataloaders.py`
- **Model checkpoint**

### Dataset ingestion

- [MSR-VTT data](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_msrvtt_retrieval.py): `dataloaders/dataloader_msrvtt_retrieval.py`
- [MSVD data](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_msvd_retrieval.py): `dataloaders/dataloader_msvd_retrieval.py`
- [LSMDC data](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_lsmdc_retrieval.py): `dataloaders/dataloader_lsmdc_retrieval.py`
- [ActivityNet data](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_activitynet_retrieval.py): `dataloaders/dataloader_activitynet_retrieval.py`
- [DiDeMo data](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/dataloader_didemo_retrieval.py): `dataloaders/dataloader_didemo_retrieval.py`
- [Frame extraction](https://github.com/arrowluo/clip4clip/blob/master/dataloaders/rawvideo_util.py): `dataloaders/rawvideo_util.py`

### Model and similarity

- [CLIP tokenizer](https://github.com/arrowluo/clip4clip/blob/master/modules/tokenization_clip.py): `modules/tokenization_clip.py`
- [CLIP4Clip model](https://github.com/arrowluo/clip4clip/blob/master/modules/modeling.py): `modules/modeling.py`
- [CLIP encoder](https://github.com/arrowluo/clip4clip/blob/master/modules/module_clip.py): `modules/module_clip.py`
- [Cross transformer](https://github.com/arrowluo/clip4clip/blob/master/modules/module_cross.py): `modules/module_cross.py`
- [Model primitives](https://github.com/arrowluo/clip4clip/blob/master/modules/until_module.py): `modules/until_module.py`
- [Model configuration](https://github.com/arrowluo/clip4clip/blob/master/modules/until_config.py): `modules/until_config.py`
- [Model file cache](https://github.com/arrowluo/clip4clip/blob/master/modules/file_utils.py): `modules/file_utils.py`

### Training support

- [Optimizer schedule](https://github.com/arrowluo/clip4clip/blob/master/modules/optimization.py): `modules/optimization.py`
- [Retrieval metrics](https://github.com/arrowluo/clip4clip/blob/master/metrics.py): `metrics.py`
- [Runtime utilities](https://github.com/arrowluo/clip4clip/blob/master/util.py): `util.py`

### Other components

- **Researcher**

## Connections

- Researcher → Retrieval runner: configures and runs (evidence: `README.md`)
- Retrieval runner → Loader factory: requests batches (evidence: `main_task_retrieval.py`)
- Loader factory → MSR-VTT data: constructs loader (evidence: `dataloaders/data_dataloaders.py`)
- Loader factory → MSVD data: constructs loader (evidence: `dataloaders/data_dataloaders.py`)
- Loader factory → LSMDC data: constructs loader (evidence: `dataloaders/data_dataloaders.py`)
- Loader factory → ActivityNet data: constructs loader (evidence: `dataloaders/data_dataloaders.py`)
- Loader factory → DiDeMo data: constructs loader (evidence: `dataloaders/data_dataloaders.py`)
- MSR-VTT data → Frame extraction: extracts frames (evidence: `dataloaders/dataloader_msrvtt_retrieval.py`)
- MSVD data → Frame extraction: extracts frames (evidence: `dataloaders/dataloader_msvd_retrieval.py`)
- LSMDC data → Frame extraction: extracts frames (evidence: `dataloaders/dataloader_lsmdc_retrieval.py`)
- ActivityNet data → Frame extraction: extracts frames (evidence: `dataloaders/dataloader_activitynet_retrieval.py`)
- DiDeMo data → Frame extraction: extracts frames (evidence: `dataloaders/dataloader_didemo_retrieval.py`)
- Retrieval runner → CLIP tokenizer: creates tokenizer (evidence: `main_task_retrieval.py`)
- Retrieval runner → CLIP4Clip model: initializes and trains (evidence: `main_task_retrieval.py`)
- CLIP4Clip model → CLIP encoder: uses encoder (evidence: `modules/modeling.py`)
- CLIP4Clip model → Cross transformer: uses transformer (evidence: `modules/modeling.py`)
- Cross transformer → Model primitives: uses layers (evidence: `modules/module_cross.py`)
- CLIP4Clip model → Model configuration: loads configuration (evidence: `modules/modeling.py`)
- Model configuration → Model file cache: resolves archives (evidence: `modules/until_config.py`)
- Cross transformer → Model file cache: resolves pretrained files (evidence: `modules/module_cross.py`)
- Retrieval runner → Optimizer schedule: updates parameters (evidence: `main_task_retrieval.py`)
- Retrieval runner → Retrieval metrics: evaluates retrieval (evidence: `main_task_retrieval.py`)
- Retrieval runner → Runtime utilities: uses runtime helpers (evidence: `main_task_retrieval.py`)
- Retrieval runner → Model checkpoint: saves or resumes (evidence: `README.md`)