How kimrass/vivit works
Main Components
• Input Data: Represent the video tensor input (with dimensions for batch, channels, frames, height, and width) as the starting point.
• Base Model: Include a component for the backbone vision transformer model that is loaded using timm (e.g., “vit_base_patch16_224”). This acts as the feature extractor.
• SpatTempoAttnViViT (which uses integrated spatio-temporal attention)
• FactorEncViViT (employs a factorised encoding scheme)
• FactorSelfAttnViViT (uses factorised self-attention)
• Preprocessing / Patch Extraction: Since the model requires video frames to be partitioned into spatial and temporal patches, add a component labeled “Preprocessing – Patch Extraction” that indicates how video inputs are divided using specified patch sizes (tempo_patch_size and spat_patch_size).
• Pooling/Classification: Represent the final steps such as “pooling” (with a mode such as “cls”) and “classification” which produce the final output (e.g., predictions over num_classes).
Architectural Patterns & Important Details
• Modular design: separation between data processing (preprocessing of video patches), feature extraction (backbone ViT), and video-specific transformer modules.
• That the design uses typical deep learning components (layers, attention modules) and relies on PyTorch and timm library integrations.
• Stay as detailed as possible: include parameter labels (like num_frames, img_size, tempo_patch_size, spat_patch_size) in the appropriate blocks to capture the data transformations involved.