# google-deepmind/videoprism architecture

> Architecture diagram of the GitHub repository google-deepmind/videoprism, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/videoprism
- Repository: https://github.com/google-deepmind/videoprism

## Overview

##### Project Type and Purpose
- VideoPrism is an open-source Python library (JAX/Flax) providing pre-trained video and video-text encoder models for a wide range of video understanding tasks (classification, retrieval, captioning, QA).
- It is NOT a full-stack application or web service, but a foundational toolkit that user code or Colab notebooks import to load checkpoints, run inference, and produce embeddings.

##### File Structure Analysis
- Top-level files:
- setup.py, requirements.txt – packaging & dependencies
- README.md – usage, model variants, and references
- LICENSE, CONTRIBUTING.md
- Package directory `videoprism/`:
- assets/ – sample video + test SentencePiece model
- colabs/ – two Jupyter notebooks demonstrating video encoder & video-text encoder flows
- encoders.py / encoders_test.py – API entrypoints: `get_model()`, `load_pretrained_weights()`
- models.py / models_test.py – Flax model architectures (ViT-based backbones, cross-modal heads)
- layers.py / layers_test.py – building blocks: attention, positional embeddings, temporal interpolation
- tokenizers.py / tokenizers_test.py – text tokenization using SentencePiece
- utils.py / utils_test.py – helper functions (normalization, data pipelines)

##### README Insights
- Core usage: user imports `videoprism.models` and `videoprism.tokenizers`, calls `get_model()`, downloads weights from Hugging Face, then `flax_model.apply()` within a `jax.jit`-decorated function.
- The library depends on JAX, Flax, and SentencePiece, and integrates with external model hosting (Hugging Face).
- Colab notebooks illustrate end-to-end flows: raw video → model → embeddings, and video↔text retrieval example.

##### Main Components
a. User Application Layer
- Python script / Colab notebook that invokes the API.
b. VideoPrism Library Layer
- encoders.py (API layer)
- models.py (Flax architectures)
- layers.py (model components)
- tokenizers.py (SentencePiece wrapper)
- utils.py (pre/post-processing)
- assets (local test data)
c. External Dependencies
- JAX & Flax frameworks
- SentencePiece model files
- Hugging Face model hubs (for downloading checkpoints)
- (Optional) Colab environment
d. Data Flow
1) Raw video files (user or assets) → utils: frame extraction & normalization
2) For video encoder: frames → models (ViT backbone + temporal embeddings) → video feature tokens
3) For video-text encoder: frames → video branch; text → tokenizers → text branch → cross-modal projection → embeddings
4) Embeddings → downstream tasks (user-defined classification, retrieval, etc.)

##### Architectural Patterns and Principles
- Modular design: clear separation of API (encoders), model definitions (models), computational primitives (layers), and preprocessing (tokenizers, utils).
- Functional programming style (JAX): pure functions, explicit state passing, `jax.jit` for compilation.
- Test coverage alongside each module to ensure reliability and facilitate extension.
- Extensibility points: adding new model variants, swapping tokenizers, integrating PyTorch support (TODO).

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "User Application Layer"
        U1["Colab: video_encoder_demo.ipynb"]:::user
        U2["Colab: video_text_demo.ipynb"]:::user
    end

    subgraph "VideoPrism Library Layer"
        Enc["videoprism.encoders"]:::core
        Models["videoprism.models"]:::core
        Layers["videoprism.layers"]:::core
        Token["videoprism.tokenizers"]:::core
        Utils["videoprism.utils"]:::core
        Assets["videoprism.assets"]:::core
    end

    subgraph "External Dependencies"
        ColabEnv["Colab Environment"]:::ext
        JAX["JAX"]:::ext
        Flax["Flax"]:::ext
        SentencePiece["SentencePiece"]:::ext
        HF["Hugging Face Checkpoints"]:::ext
    end

    U1 -->|"invoke get_model()/load_pretrained_weights()"| Enc
    U2 -->|"invoke get_model()/load_pretrained_weights()"| Enc
    Enc -->|"calls Flax architectures"| Models
    Models -->|"uses computational primitives"| Layers
    Token -->|"uses SentencePiece model"| SentencePiece
    Token -->|"provides text tokens to"| Models
    Utils -->|"preprocesses raw video/text"| Enc
    Assets -->|"sample data to"| Utils
    Enc -->|"downloads weights from"| HF
    Models -->|"downloads weights from"| HF
    ColabEnv --> U1
    ColabEnv --> U2

    classDef user fill:#D6EAF8,stroke:#1B4F72
    classDef core fill:#AED6F1,stroke:#154360
    classDef ext fill:#A9DFBF,stroke:#145A32

    click U1 "https://github.com/google-deepmind/videoprism/blob/main/videoprism/colabs/videoprism_video_encoder_demo.ipynb"
    click U2 "https://github.com/google-deepmind/videoprism/blob/main/videoprism/colabs/videoprism_video_text_demo.ipynb"
    click Enc "https://github.com/google-deepmind/videoprism/blob/main/videoprism/encoders.py"
    click Models "https://github.com/google-deepmind/videoprism/blob/main/videoprism/models.py"
    click Layers "https://github.com/google-deepmind/videoprism/blob/main/videoprism/layers.py"
    click Token "https://github.com/google-deepmind/videoprism/blob/main/videoprism/tokenizers.py"
    click Utils "https://github.com/google-deepmind/videoprism/blob/main/videoprism/utils.py"
    click Assets "https://github.com/google-deepmind/videoprism/tree/main/videoprism/assets/testdata/"
```
