# google-research/vision_transformer architecture

> Architecture diagram of the GitHub repository google-research/vision_transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-research/vision_transformer
- Repository: https://github.com/google-research/vision_transformer

## Overview

##### Project type and purpose
- This is an open-source machine-learning toolkit for fine-tuning and evaluating Vision Transformer (ViT), MLP-Mixer, and LiT models in JAX/Flax.
- It provides a command-line interface (via `vit_jax.main` and `vit_jax.train`), data pipelines, model definitions, configuration management, checkpoint I/O, and Jupyter notebooks for interactive exploration.

##### File structure
- Top-level directories and files:
- `vit_jax/` – core python package
- `model_cards/` – documentation of LiT model performances
- `*.ipynb` – Colab-ready notebooks for interactive demos
- `setup.py`/`version.py` – packaging
- `.github/workflows` – CI build/test
- Within `vit_jax/`:
- `configs/` – ConfigDicts for different models, datasets, augmentation regimes
- `input_pipeline.py` – data ingestion, TFDS or local JPEG
- `preprocess.py` – image transforms (patching, normalization)
- `models*.py` – model definitions: ViT, Mixer, hybrids, ResNet backbones
- `train.py` / `inference_time.py` – training and benchmarking loops
- `checkpoint.py` – save/load `.npz` weights (GCS integration)
- `utils.py` / `test_utils.py` – helpers, testing scaffolding
- `main.py` – CLI entrypoint wiring configs → training/eval
- `*_test.py` – unit tests for pipeline, models, checkpoint logic

##### Main components
a. Configuration layer
– `configs/`: load hyperparameters, model choice, dataset specs via ml_collections.ConfigDict
b. Data ingestion & preprocessing
– `input_pipeline.py` → `preprocess.py`
– Interfaces: TensorFlow Datasets, Google Cloud Storage for JPEGs
c. Model assembly
– `models_*.py`: constructs JAX/Flax modules (ViT, Mixer, ResNet hybrid)
d. Training loop
– `train.py`: forward & backward passes, optimizer (AdamW), gradient accumulation, multi-device parallelism (JAX pmap)
– `inference_time.py`: benchmarking throughput
e. Checkpoint management
– `checkpoint.py`: reading/writing model weights to local disk or GCS buckets
f. Evaluation & logging
– summary writers, TensorBoard hooks
g. CLI & Notebooks
– `main.py`: glues configs, training, evaluation
– `*.ipynb`: interactive demos, model exploration
h. External dependencies
– JAX, Flax, TensorFlow Datasets, Cloud Storage, TPU/GPU devices

##### Relationships and interactions
- Startup: user invokes Python module (`python -m vit_jax.main` or `train.py`)
- Configuration loader reads `configs/*.py` → populates a ConfigDict
- Input pipeline pulls raw data from TFDS or local storage → applies transforms
- Model factory builds a JAX/Flax model instance according to config
- Training loop dispatches data → model → loss → optimizer → updates
- Checkpoint manager periodically saves weights; on resume loads weights
- Evaluation loop computes accuracy/metrics and writes logs
- Notebooks import underlying modules to orchestrate the above interactively

##### Architectural patterns & design principles
- Modular “plug-and-play” configurations (ml_collections)
- Separation of concerns: data, model, training, I/O, CLI
- Multi-host/multi-device abstraction via JAX pmap
- Test-driven for core pipeline and model correctness
- Notebook wrappers for reproducibility and education

##### Detail and extensibility
- Test harnesses (unit tests) branching off core modules
- Notebook layer invoking the same core pipelines

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        CM["Config Manager"]:::core
        C1["augreg.py"]:::core
        C2["common.py"]:::core
        C3["inference_time.py"]:::core
        C4["mixer_base16_cifar10.py"]:::core
        C5["models.py"]:::core
        C6["vit.py"]:::core
        CM --> C1
        CM --> C2
        CM --> C3
        CM --> C4
        CM --> C5
        CM --> C6
    end

    %% Data Pipeline Layer
    subgraph "Data Pipeline"
        direction TB
        IP["Data Ingestion Pipeline"]:::data
        PP["Preprocessing Module"]:::data
        IP --> PP
    end

    %% Model Factory Layer
    subgraph "Model Factory"
        direction TB
        MF["Model Factory"]:::model
        MV["models.py / models_vit.py"]:::model
        MM["models_mixer.py"]:::model
        ML["models_lit.py"]:::model
        MR["models_resnet.py"]:::model
        MF --> MV
        MF --> MM
        MF --> ML
        MF --> MR
    end

    %% Training & Inference Layer
    subgraph "Training & Inference"
        direction TB
        TR["Training Loop"]:::training
        IN["Inference / Benchmarking"]:::training
        TR --> IN
    end

    %% Checkpoint Management
    CP["Checkpointer"]:::core

    %% Evaluation & Logging
    EV["Evaluator & Logging"]:::training

    %% CLI & Notebooks
    subgraph "CLI & Notebooks"
        direction TB
        CLI["CLI Entrypoint"]:::core
        NB["Jupyter Notebooks"]:::core
    end

    %% Utilities & Tests
    subgraph "Utilities & Tests"
        direction TB
        UT["Utils Module"]:::core
        TU["Test Utils"]:::core
        PT["Preprocess Tests"]:::core
        TT["Train Tests"]:::core
        ITT["Inference Tests"]:::core
        CT["Checkpoint Tests"]:::core
    end

    %% External Dependencies
    subgraph "External Services"
        direction TB
        TFDS["TensorFlow Datasets"]:::external
        GCS["Google Cloud Storage"]:::external
        DEV["TPU/GPU Devices"]:::external
    end

    %% Connections
    CM --> IP
    CM --> MF
    CM --> TR
    IP -->|"raw data"| TFDS
    IP -->|"JPEGs"| GCS
    PP --> TR
    MF --> TR
    TR -->|loads/saves| CP
    CP -->|load on resume| TR
    TR --> EV
    EV -->|"logs"| TFDS
    CLI --> CM
    CLI --> TR
    CLI --> EV
    NB --> CM
    NB --> TR
    NB --> EV
    UT --> IP
    UT --> PP
    UT --> MF
    UT --> TR
    UT --> CP
    PT --> PP
    TT --> TR
    ITT --> IN
    CT --> CP
    DEV --> TR
    DEV -.->|execution| IN

    %% Click Events
    click CM "https://github.com/google-research/vision_transformer/tree/main/vit_jax/configs/"
    click C1 "https://github.com/google-research/vision_transformer/blob/main/vit_jax/configs/augreg.py"
    click C2 "https://github.com/google-research/vision_transformer/blob/main/vit_jax/configs/common.py"
    click C3 "https://github.com/google-research/vision_transformer/blob/main/vit_jax/configs/inference_time.py"
    click C4 "https://github.com/google-research/vision_transformer/blob/main/vit_jax/configs/mixer_base16_cifar10.py"
    click C5 "https://github.com/google-research/vision_transformer/blob/main/vit_jax/configs/models.py"
    click C6 "https://github.com/google-research/vision_transformer/blob/main/vit_jax/configs/vit.py"
    click IP "https://github.com/google-research/vision_transformer/blob/main/vit_jax/input_pipeline.py"
    click PP "https://github.com/google-research/vision_transformer/blob/main/vit_jax/preprocess.py"
    click MV "https://github.com/google-research/vision_transformer/blob/main/vit_jax/models.py"
    click MM "https://github.com/google-research/vision_transformer/blob/main/vit_jax/models_mixer.py"
    click ML "https://github.com/google-research/vision_transformer/blob/main/vit_jax/models_lit.py"
    click MR "https://github.com/google-research/vision_transformer/blob/main/vit_jax/models_resnet.py"
    click TR "https://github.com/google-research/vision_transformer/blob/main/vit_jax/train.py"
    click TT "https://github.com/google-research/vision_transformer/blob/main/vit_jax/train_test.py"
    click IN "https://github.com/google-research/vision_transformer/blob/main/vit_jax/inference_time.py"
    click ITT "https://github.com/google-research/vision_transformer/blob/main/vit_jax/inference_time_test.py"
    click CP "https://github.com/google-research/vision_transformer/blob/main/vit_jax/checkpoint.py"
    click CT "https://github.com/google-research/vision_transformer/blob/main/vit_jax/checkpoint_test.py"
    click CLI "https://github.com/google-research/vision_transformer/blob/main/vit_jax/main.py"
    click UT "https://github.com/google-research/vision_transformer/blob/main/vit_jax/utils.py"
    click TU "https://github.com/google-research/vision_transformer/blob/main/vit_jax/test_utils.py"
    click NB "https://github.com/google-research/vision_transformer/blob/main/lit.ipynb"
    click NB "https://github.com/google-research/vision_transformer/blob/main/vit_jax.ipynb"
    click NB "https://github.com/google-research/vision_transformer/blob/main/vit_jax_augreg.ipynb"
    click setup "https://github.com/google-research/vision_transformer/blob/main/setup.py"
    click version "https://github.com/google-research/vision_transformer/blob/main/version.py"
    click workflow "https://github.com/google-research/vision_transformer/blob/main/.github/workflows/build.yml"

    %% Styles
    classDef core fill:#add8e6,stroke:#333,stroke-width:1px
    classDef data fill:#90ee90,stroke:#333,stroke-width:1px
    classDef model fill:#ffcc99,stroke:#333,stroke-width:1px
    classDef training fill:#d8bfd8,stroke:#333,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
```
