# kimrass/vivit architecture

> Architecture diagram of the GitHub repository kimrass/vivit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kimrass/vivit
- Repository: https://github.com/kimrass/vivit

## Overview

##### Main Components
- Input Data: Represent the video tensor input (with dimensions for batch, channels, frames, height, and width) as the starting point.
- Base Model: Include a component for the backbone vision transformer model that is loaded using timm (e.g., “vit_base_patch16_224”). This acts as the feature extractor.
- SpatTempoAttnViViT (which uses integrated spatio-temporal attention)
- FactorEncViViT (employs a factorised encoding scheme)
- FactorSelfAttnViViT (uses factorised self-attention)
- Preprocessing / Patch Extraction: Since the model requires video frames to be partitioned into spatial and temporal patches, add a component labeled “Preprocessing – Patch Extraction” that indicates how video inputs are divided using specified patch sizes (tempo_patch_size and spat_patch_size).
- Pooling/Classification: Represent the final steps such as “pooling” (with a mode such as “cls”) and “classification” which produce the final output (e.g., predictions over num_classes).

##### Architectural Patterns & Important Details
- Modular design: separation between data processing (preprocessing of video patches), feature extraction (backbone ViT), and video-specific transformer modules.
- That the design uses typical deep learning components (layers, attention modules) and relies on PyTorch and timm library integrations.

- Stay as detailed as possible: include parameter labels (like num_frames, img_size, tempo_patch_size, spat_patch_size) in the appropriate blocks to capture the data transformations involved.

## Diagram (Mermaid)

```mermaid
flowchart TD
    inputVideo(("Input Video")):::data
    preproc["Preprocessing – Patch Extraction<br/>(tempo_patch_size, spat_patch_size)"]:::preproc
    backbone["Base Vision Transformer (timm integration)<br/>(e.g., vit_base_patch16_224)"]:::module
    pooling["Pooling/Classification<br/>(cls pooling, num_classes)"]:::output
    libraries["External: PyTorch & timm"]:::external

    subgraph "ViViT Modules"
        direction TB
        module1["SpatTempoAttnViViT Module"]:::module
        module2["FactorEncViViT Module"]:::module
        module3["FactorSelfAttnViViT Module"]:::module
    end

    subgraph "Research Papers"
        direction TB
        research1["Video Vision Transformers for Violence Detection"]:::doc
        research2["ViViT: A Video Vision Transformer"]:::doc
    end

    %% Data Flow
    inputVideo --> preproc
    preproc --> backbone
    backbone --> module1
    backbone --> module2
    backbone --> module3
    module1 --> pooling
    module2 --> pooling
    module3 --> pooling

    %% External Dependencies
    backbone --- libraries

    %% Research Linkage (informing the design)
    research1 --- backbone
    research2 --- backbone

    %% Click Events
    click inputVideo "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click preproc "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click backbone "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click module1 "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click module2 "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click module3 "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click pooling "https://github.com/kimrass/vivit/blob/main/vivit.py"
    click research1 "https://github.com/kimrass/vivit/blob/main/papers/video_vision_transformers_for_violence_detection.pdf"
    click research2 "https://github.com/kimrass/vivit/blob/main/papers/vivit_a_video_vision_transformer.pdf"

    %% Styles
    classDef data fill:#cce5ff,stroke:#333,stroke-width:2px
    classDef preproc fill:#d4edda,stroke:#333,stroke-width:2px
    classDef module fill:#f8d7da,stroke:#333,stroke-width:2px
    classDef output fill:#fff3cd,stroke:#333,stroke-width:2px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:2px
    classDef doc fill:#f0d9ff,stroke:#333,stroke-width:2px
```
