How rishikksh20/vivit-pytorch works
1. This repository is a small, self-contained PyTorch model implementation rather than a full application. Its purpose is to define and run an unofficial ViViT video classification model.
2. The core runtime boundary is the model API exposed from vivit.py. From the README usage, the model accepts a 5D video tensor shaped like [B, T, C, H, W] and returns class logits shaped [B, num_classes].
3. vivit.py is the main architecture file. It likely contains the top-level ViViT class, model wiring, and the forward pass that transforms video frames into a classification output.
4. module.py appears to hold reusable building blocks shared by the model, such as transformer layers, attention blocks, patch embedding logic, and other helper modules used by vivit.py. This is the likely internal separation between “model assembly” and “primitive components.”
5. The model is a Video Vision Transformer variant, so the data flow is conceptually: input video → frame/patch tokenization → transformer-based spatiotemporal processing → pooled representation → classification head. The README example confirms the final output is a per-class prediction tensor.
6. The implementation status matters architecturally: only “Model 2” from the ViViT paper is implemented, while Models 3 and 4 are not. That means this repo represents a partial reproduction of the paper’s model family, not a complete ViViT suite.
7. The project is oriented around a single inference/training entry point in Python, with PyTorch as the underlying runtime. The usage snippet also depends on NumPy for parameter counting, but NumPy is not part of the model architecture itself.
8. The repository has no service layer, dataset pipeline, or training framework visible in the tree. There are no APIs, job queues, databases, or deployment components—just model code and documentation.
9. assets/ is documentation support only; the included model image helps explain the architecture visually but does not participate in execution.
10. The codebase inherits design ideas from external implementations: lucidrains’ vit-pytorch for base ViT mechanics and STAM-pytorch for some Model 2 logic. Those dependencies shape the implementation style but are vendored/adapted into this repo rather than consumed as a runtime service.
11. For an architecture diagram, the most accurate top-level boxes are: “video tensor input,” “ViViT model core,” “shared transformer/attention modules,” and “classification output.” Any diagram beyond that should stay faithful to the fact that this is a single-model library with one implemented variant.
Main entry points: vivit.py vivit.py · README.md README.md · module.py module.py · assets assets/
Components 12
Docs
Documentation and explanatory assets.
ViViT Core
Top-level model assembly and execution path.
- Video tensor · 5D batch of videos shaped like [B, T, C, H, W].
- vivit.py · Main ViViT model definition and forward pass.
vivit.py - Tokenize · Converts frames into patch tokens for transformer processing.
- Transformer stack · Spatiotemporal processing for the implemented ViViT Model 2 variant.
- Class head · Pools the learned representation and maps it to class logits.
- Class logits · Per-class prediction tensor shaped like [B, num_classes].
- Model 2 · The implemented ViViT paper variant in this repository.
Shared Modules
Reusable transformer and embedding building blocks.
- module.py · Reusable model components shared by the ViViT assembly.
module.py - Patch embed · Patch and frame embedding logic used by the model.
- Attention block · Attention and feed-forward primitives for ViViT layers.
Connections 14
- README.md → vivit.py: describes usage · Documentation points to the main model API.
- README.md → Video tensor: example input · Shows the expected video tensor shape.
- Video tensor → vivit.py: feeds · Video batches are passed into the model entry point.
- vivit.py → Tokenize: builds · The model first converts frames into token sequences.
- Tokenize → Patch embed: uses · Tokenization relies on patch embedding primitives.
- vivit.py → Transformer stack: runs · Token sequences are processed by the transformer core.
- Transformer stack → Attention block: uses · Core layers are composed from shared attention blocks.
- Attention block → module.py: defined in · Attention and transformer primitives live in the shared module file.
- Patch embed → module.py: defined in · Embedding utilities are implemented in the shared module file.
- vivit.py → Model 2: implements · This repository focuses on the Model 2 variant of ViViT.
- Model 2 → Class head: ends at · The implemented variant produces class predictions through a head.
- Transformer stack → Class head: feeds · Encoded representations are pooled before classification.
- Class head → Class logits: returns · Final output is per-class logits.
- assets/ → README.md: illustrates · Supporting images help explain the documented architecture.