How arrowluo/clip4clip works
CLIP4Clip trains and evaluates video–text retrieval models across five benchmark datasets. The main workflow is to select a dataset loader, batch caption/video examples, encode them with CLIP4Clip, optimize retrieval training, and evaluate retrieval metrics. The model supports parameter-free, sequential, and tight similarity approaches, with configurable frame sampling and 2D/3D patches. Edges follow the excerpts, SOURCE INDEX references, or README; unsampled implementation details are not inferred.
Main entry points: main_task_retrieval.py Retrieval runner · data_dataloaders.py Loader factory · rawvideo_util.py Frame extraction · modeling.py CLIP4Clip model
Components 20
Retrieval workflow
- Retrieval runner
main_task_retrieval.py - Loader factory
dataloaders/data_dataloaders.py - Model checkpoint
Dataset ingestion
- MSR-VTT data
dataloaders/dataloader_msrvtt_retrieval.py - MSVD data
dataloaders/dataloader_msvd_retrieval.py - LSMDC data
dataloaders/dataloader_lsmdc_retrieval.py - ActivityNet data
dataloaders/dataloader_activitynet_retrieval.py - DiDeMo data
dataloaders/dataloader_didemo_retrieval.py - Frame extraction
dataloaders/rawvideo_util.py
Model and similarity
- CLIP tokenizer
modules/tokenization_clip.py - CLIP4Clip model
modules/modeling.py - CLIP encoder
modules/module_clip.py - Cross transformer
modules/module_cross.py - Model primitives
modules/until_module.py - Model configuration
modules/until_config.py - Model file cache
modules/file_utils.py
Training support
- Optimizer schedule
modules/optimization.py - Retrieval metrics
metrics.py - Runtime utilities
util.py
Other components
- Researcher
Connections 24
- Researcher → Retrieval runner: configures and runs
README.md - Retrieval runner → Loader factory: requests batches
main_task_retrieval.py - Loader factory → MSR-VTT data: constructs loader
dataloaders/data_dataloaders.py - Loader factory → MSVD data: constructs loader
dataloaders/data_dataloaders.py - Loader factory → LSMDC data: constructs loader
dataloaders/data_dataloaders.py - Loader factory → ActivityNet data: constructs loader
dataloaders/data_dataloaders.py - Loader factory → DiDeMo data: constructs loader
dataloaders/data_dataloaders.py - MSR-VTT data → Frame extraction: extracts frames
dataloaders/dataloader_msrvtt_retrieval.py - MSVD data → Frame extraction: extracts frames
dataloaders/dataloader_msvd_retrieval.py - LSMDC data → Frame extraction: extracts frames
dataloaders/dataloader_lsmdc_retrieval.py - ActivityNet data → Frame extraction: extracts frames
dataloaders/dataloader_activitynet_retrieval.py - DiDeMo data → Frame extraction: extracts frames
dataloaders/dataloader_didemo_retrieval.py - Retrieval runner → CLIP tokenizer: creates tokenizer
main_task_retrieval.py - Retrieval runner → CLIP4Clip model: initializes and trains
main_task_retrieval.py - CLIP4Clip model → CLIP encoder: uses encoder
modules/modeling.py - CLIP4Clip model → Cross transformer: uses transformer
modules/modeling.py - Cross transformer → Model primitives: uses layers
modules/module_cross.py - CLIP4Clip model → Model configuration: loads configuration
modules/modeling.py - Model configuration → Model file cache: resolves archives
modules/until_config.py - Cross transformer → Model file cache: resolves pretrained files
modules/module_cross.py - Retrieval runner → Optimizer schedule: updates parameters
main_task_retrieval.py - Retrieval runner → Retrieval metrics: evaluates retrieval
main_task_retrieval.py - Retrieval runner → Runtime utilities: uses runtime helpers
main_task_retrieval.py - Retrieval runner → Model checkpoint: saves or resumes
README.md