How openai/clip works
CLIP is a Python library for comparing images and natural-language text with a shared neural model. The principal workflow is to load a checkpoint and image transform, preprocess an image, tokenize candidate text, then encode both modalities and score their cosine similarity. The README also documents zero-shot label ranking and linear-probe use of image features; these are downstream usage patterns, not separate library runtime stages. Source excerpts establish the loader, tokenizer, and model structure; unsampled implementation details are omitted.
Main entry points: clip.py CLIP API · model.py CLIP model · simple_tokenizer.py BPE tokenizer · hubconf.py Torch Hub
Components 14
Input preparation
- Image transform
clip/clip.py - Text tokenization
clip/clip.py - BPE tokenizer
clip/simple_tokenizer.py - BPE merges
clip/bpe_simple_vocab_16e6.txt.gz
Model inference
- Model builder
clip/model.py - CLIP model
clip/model.py - Vision encoder
clip/model.py - Text encoder
clip/model.py - Similarity scores
clip/model.py
Model weights
- Checkpoint loader
clip/clip.py - Model checkpoint
Other components
- Application
Connections 15
- Application → CLIP API: uses
README.md - Application → Image transform: supplies image
README.md - Application → Text tokenization: supplies text
README.md - Torch Hub → CLIP API: delegates loading
hubconf.py - CLIP API → Checkpoint loader: loads model
clip/clip.py - CLIP API → Text tokenization: tokenizes text
clip/clip.py - CLIP API → Image transform: creates transform
clip/clip.py - Text tokenization → BPE tokenizer: encodes text
clip/clip.py - BPE tokenizer → BPE merges: reads merges
clip/simple_tokenizer.py - Checkpoint loader → Model checkpoint: downloads or reads
clip/clip.py - Checkpoint loader → Model builder: builds model
clip/clip.py - CLIP model → Vision encoder: encodes images
clip/model.py - CLIP model → Text encoder: encodes tokens
clip/model.py - CLIP model → Similarity scores: scores similarities
README.md - Application → Similarity scores: uses scores
README.md