# openai/clip architecture

> Architecture diagram of the GitHub repository openai/clip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/clip
- Repository: https://github.com/openai/clip

## Overview

CLIP is a Python library for comparing images and natural-language text with a shared neural model. The principal workflow is to load a checkpoint and image transform, preprocess an image, tokenize candidate text, then encode both modalities and score their cosine similarity. The README also documents zero-shot label ranking and linear-probe use of image features; these are downstream usage patterns, not separate library runtime stages. Source excerpts establish the loader, tokenizer, and model structure; unsampled implementation details are omitted.

Main entry points: [`clip/clip.py`](https://github.com/openai/clip/blob/main/clip/clip.py) (CLIP API), [`clip/model.py`](https://github.com/openai/clip/blob/main/clip/model.py) (CLIP model), [`clip/simple_tokenizer.py`](https://github.com/openai/clip/blob/main/clip/simple_tokenizer.py) (BPE tokenizer), [`hubconf.py`](https://github.com/openai/clip/blob/main/hubconf.py) (Torch Hub)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_access["Library access"]
  node_api["CLIP API<br/>[clip.py]"]
  node_hub["Torch Hub<br/>[hubconf.py]"]
end

subgraph group_inputs["Input preparation"]
  node_preprocess["Image transform<br/>[clip.py]"]
  node_tokenize["Text tokenization<br/>[clip.py]"]
  node_bpe["BPE tokenizer"]
  node_vocab["BPE merges"]
end

subgraph group_inference["Model inference"]
  node_builder["Model builder<br/>[model.py]"]
  node_clipmodel["CLIP model<br/>[model.py]"]
  node_vision["Vision encoder<br/>[model.py]"]
  node_text["Text encoder<br/>[model.py]"]
  node_similarity["Similarity scores<br/>[model.py]"]
end

subgraph group_weights["Model weights"]
  node_loader["Checkpoint loader<br/>[clip.py]"]
  node_checkpoint["Model checkpoint"]
end

node_caller(("Application"))

node_caller -->|"uses"| node_api
node_caller -->|"supplies image"| node_preprocess
node_caller -->|"supplies text"| node_tokenize
node_hub -->|"delegates loading"| node_api
node_api -->|"loads model"| node_loader
node_api -->|"tokenizes text"| node_tokenize
node_api -->|"creates transform"| node_preprocess
node_tokenize -->|"encodes text"| node_bpe
node_bpe -->|"reads merges"| node_vocab
node_loader -->|"downloads or reads"| node_checkpoint
node_loader -->|"builds model"| node_builder
node_clipmodel -->|"encodes images"| node_vision
node_clipmodel -->|"encodes tokens"| node_text
node_clipmodel -->|"scores similarities"| node_similarity
node_caller -->|"uses scores"| node_similarity

click node_api "https://github.com/openai/clip/blob/main/clip/clip.py"
click node_hub "https://github.com/openai/clip/blob/main/hubconf.py"
click node_preprocess "https://github.com/openai/clip/blob/main/clip/clip.py"
click node_tokenize "https://github.com/openai/clip/blob/main/clip/clip.py"
click node_bpe "https://github.com/openai/clip/blob/main/clip/simple_tokenizer.py"
click node_vocab "https://github.com/openai/clip/blob/main/clip/bpe_simple_vocab_16e6.txt.gz"
click node_loader "https://github.com/openai/clip/blob/main/clip/clip.py"
click node_builder "https://github.com/openai/clip/blob/main/clip/model.py"
click node_clipmodel "https://github.com/openai/clip/blob/main/clip/model.py"
click node_vision "https://github.com/openai/clip/blob/main/clip/model.py"
click node_text "https://github.com/openai/clip/blob/main/clip/model.py"
click node_similarity "https://github.com/openai/clip/blob/main/clip/model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_hub toneBlue
class node_preprocess,node_tokenize,node_bpe,node_vocab toneAmber
class node_builder,node_clipmodel,node_vision,node_text,node_similarity toneMint
class node_loader,node_checkpoint toneRose
class node_caller toneIndigo
```

## Components

### Library access

- [CLIP API](https://github.com/openai/clip/blob/main/clip/clip.py): `clip/clip.py`
- [Torch Hub](https://github.com/openai/clip/blob/main/hubconf.py): `hubconf.py`

### Input preparation

- [Image transform](https://github.com/openai/clip/blob/main/clip/clip.py): `clip/clip.py`
- [Text tokenization](https://github.com/openai/clip/blob/main/clip/clip.py): `clip/clip.py`
- [BPE tokenizer](https://github.com/openai/clip/blob/main/clip/simple_tokenizer.py): `clip/simple_tokenizer.py`
- [BPE merges](https://github.com/openai/clip/blob/main/clip/bpe_simple_vocab_16e6.txt.gz): `clip/bpe_simple_vocab_16e6.txt.gz`

### Model inference

- [Model builder](https://github.com/openai/clip/blob/main/clip/model.py): `clip/model.py`
- [CLIP model](https://github.com/openai/clip/blob/main/clip/model.py): `clip/model.py`
- [Vision encoder](https://github.com/openai/clip/blob/main/clip/model.py): `clip/model.py`
- [Text encoder](https://github.com/openai/clip/blob/main/clip/model.py): `clip/model.py`
- [Similarity scores](https://github.com/openai/clip/blob/main/clip/model.py): `clip/model.py`

### Model weights

- [Checkpoint loader](https://github.com/openai/clip/blob/main/clip/clip.py): `clip/clip.py`
- **Model checkpoint**

### Other components

- **Application**

## Connections

- Application → CLIP API: uses (evidence: `README.md`)
- Application → Image transform: supplies image (evidence: `README.md`)
- Application → Text tokenization: supplies text (evidence: `README.md`)
- Torch Hub → CLIP API: delegates loading (evidence: `hubconf.py`)
- CLIP API → Checkpoint loader: loads model (evidence: `clip/clip.py`)
- CLIP API → Text tokenization: tokenizes text (evidence: `clip/clip.py`)
- CLIP API → Image transform: creates transform (evidence: `clip/clip.py`)
- Text tokenization → BPE tokenizer: encodes text (evidence: `clip/clip.py`)
- BPE tokenizer → BPE merges: reads merges (evidence: `clip/simple_tokenizer.py`)
- Checkpoint loader → Model checkpoint: downloads or reads (evidence: `clip/clip.py`)
- Checkpoint loader → Model builder: builds model (evidence: `clip/clip.py`)
- CLIP model → Vision encoder: encodes images (evidence: `clip/model.py`)
- CLIP model → Text encoder: encodes tokens (evidence: `clip/model.py`)
- CLIP model → Similarity scores: scores similarities (evidence: `README.md`)
- Application → Similarity scores: uses scores (evidence: `README.md`)
