# rishikksh20/vivit-pytorch architecture

> Architecture diagram of the GitHub repository rishikksh20/vivit-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishikksh20/vivit-pytorch
- Repository: https://github.com/rishikksh20/vivit-pytorch

## Overview

1. This repository is a small, self-contained PyTorch model implementation rather than a full application. Its purpose is to define and run an unofficial ViViT video classification model.

2. The core runtime boundary is the model API exposed from `vivit.py`. From the README usage, the model accepts a 5D video tensor shaped like `[B, T, C, H, W]` and returns class logits shaped `[B, num_classes]`.

3. `vivit.py` is the main architecture file. It likely contains the top-level `ViViT` class, model wiring, and the forward pass that transforms video frames into a classification output.

4. `module.py` appears to hold reusable building blocks shared by the model, such as transformer layers, attention blocks, patch embedding logic, and other helper modules used by `vivit.py`. This is the likely internal separation between “model assembly” and “primitive components.”

5. The model is a Video Vision Transformer variant, so the data flow is conceptually: input video → frame/patch tokenization → transformer-based spatiotemporal processing → pooled representation → classification head. The README example confirms the final output is a per-class prediction tensor.

6. The implementation status matters architecturally: only “Model 2” from the ViViT paper is implemented, while Models 3 and 4 are not. That means this repo represents a partial reproduction of the paper’s model family, not a complete ViViT suite.

7. The project is oriented around a single inference/training entry point in Python, with PyTorch as the underlying runtime. The usage snippet also depends on NumPy for parameter counting, but NumPy is not part of the model architecture itself.

8. The repository has no service layer, dataset pipeline, or training framework visible in the tree. There are no APIs, job queues, databases, or deployment components—just model code and documentation.

9. `assets/` is documentation support only; the included model image helps explain the architecture visually but does not participate in execution.

10. The codebase inherits design ideas from external implementations: lucidrains’ `vit-pytorch` for base ViT mechanics and `STAM-pytorch` for some Model 2 logic. Those dependencies shape the implementation style but are vendored/adapted into this repo rather than consumed as a runtime service.

11. For an architecture diagram, the most accurate top-level boxes are: “video tensor input,” “ViViT model core,” “shared transformer/attention modules,” and “classification output.” Any diagram beyond that should stay faithful to the fact that this is a single-model library with one implemented variant.

Main entry points: [`vivit.py`](https://github.com/rishikksh20/vivit-pytorch/blob/master/vivit.py) (vivit.py), [`README.md`](https://github.com/rishikksh20/vivit-pytorch/blob/master/README.md) (README.md), [`module.py`](https://github.com/rishikksh20/vivit-pytorch/blob/master/module.py) (module.py), [`assets`](https://github.com/rishikksh20/vivit-pytorch/tree/master/assets) (assets/)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_docs["Docs"]
  node_assets["assets/<br/>docs asset"]
  node_readme["README.md<br/>usage docs<br/>[README.md]"]
end

subgraph group_core["ViViT Core"]
  node_video_input(("Video tensor<br/>input"))
  node_vivit_py["vivit.py<br/>model entry<br/>[vivit.py]"]
  node_tokenize["Tokenize<br/>embedding stage"]
  node_spatiotemporal["Transformer stack<br/>encoder"]
  node_head["Class head<br/>classifier"]
  node_logits(("Class logits<br/>output"))
  node_model2{{"Model 2<br/>implemented variant"}}
end

subgraph group_primitives["Shared Modules"]
  node_module_py["module.py<br/>shared blocks<br/>[module.py]"]
  node_patch_embed["Patch embed<br/>embedding block"]
  node_attention["Attention block<br/>transformer block"]
end

node_readme -->|"describes usage"| node_vivit_py
node_readme -.->|"example input"| node_video_input
node_video_input -->|"feeds"| node_vivit_py
node_vivit_py -->|"builds"| node_tokenize
node_tokenize -->|"uses"| node_patch_embed
node_vivit_py -->|"runs"| node_spatiotemporal
node_spatiotemporal -->|"uses"| node_attention
node_attention -->|"defined in"| node_module_py
node_patch_embed -->|"defined in"| node_module_py
node_vivit_py -->|"implements"| node_model2
node_model2 -->|"ends at"| node_head
node_spatiotemporal -->|"feeds"| node_head
node_head -->|"returns"| node_logits
node_assets -.->|"illustrates"| node_readme

click node_vivit_py "https://github.com/rishikksh20/vivit-pytorch/blob/master/vivit.py"
click node_module_py "https://github.com/rishikksh20/vivit-pytorch/blob/master/module.py"
click node_assets "https://github.com/rishikksh20/vivit-pytorch/tree/master/assets"
click node_readme "https://github.com/rishikksh20/vivit-pytorch/blob/master/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_assets,node_readme toneBlue
class node_video_input,node_vivit_py,node_tokenize,node_spatiotemporal,node_head,node_logits,node_model2 toneAmber
class node_module_py,node_patch_embed,node_attention toneMint
```

## Components

### Docs

Documentation and explanatory assets.

- [assets/](https://github.com/rishikksh20/vivit-pytorch/tree/master/assets): docs asset · Illustration files that explain the architecture visually. · `assets`
- [README.md](https://github.com/rishikksh20/vivit-pytorch/blob/master/README.md): usage docs · Usage example and high-level description of the model. · `README.md`

### ViViT Core

Top-level model assembly and execution path.

- **Video tensor**: input · 5D batch of videos shaped like [B, T, C, H, W].
- [vivit.py](https://github.com/rishikksh20/vivit-pytorch/blob/master/vivit.py): model entry · Main ViViT model definition and forward pass. · `vivit.py`
- **Tokenize**: embedding stage · Converts frames into patch tokens for transformer processing.
- **Transformer stack**: encoder · Spatiotemporal processing for the implemented ViViT Model 2 variant.
- **Class head**: classifier · Pools the learned representation and maps it to class logits.
- **Class logits**: output · Per-class prediction tensor shaped like [B, num_classes].
- **Model 2**: implemented variant · The implemented ViViT paper variant in this repository.

### Shared Modules

Reusable transformer and embedding building blocks.

- [module.py](https://github.com/rishikksh20/vivit-pytorch/blob/master/module.py): shared blocks · Reusable model components shared by the ViViT assembly. · `module.py`
- **Patch embed**: embedding block · Patch and frame embedding logic used by the model.
- **Attention block**: transformer block · Attention and feed-forward primitives for ViViT layers.

## Connections

- README.md → vivit.py: describes usage · Documentation points to the main model API.
- README.md → Video tensor: example input · Shows the expected video tensor shape.
- Video tensor → vivit.py: feeds · Video batches are passed into the model entry point.
- vivit.py → Tokenize: builds · The model first converts frames into token sequences.
- Tokenize → Patch embed: uses · Tokenization relies on patch embedding primitives.
- vivit.py → Transformer stack: runs · Token sequences are processed by the transformer core.
- Transformer stack → Attention block: uses · Core layers are composed from shared attention blocks.
- Attention block → module.py: defined in · Attention and transformer primitives live in the shared module file.
- Patch embed → module.py: defined in · Embedding utilities are implemented in the shared module file.
- vivit.py → Model 2: implements · This repository focuses on the Model 2 variant of ViViT.
- Model 2 → Class head: ends at · The implemented variant produces class predictions through a head.
- Transformer stack → Class head: feeds · Encoded representations are pooled before classification.
- Class head → Class logits: returns · Final output is per-class logits.
- assets/ → README.md: illustrates · Supporting images help explain the documented architecture.
