# google-research/magvit architecture

> Architecture diagram of the GitHub repository google-research/magvit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-research/magvit
- Repository: https://github.com/google-research/magvit

## Overview

##### Main components
- Command‐Line Interface (CLI) / Entry Point
– videogvt/main.py
– Reads CLI args, loads a config, dispatches to training, evaluation, or sampling.
- Configuration Layer
– videogvt/configs/*.py
– Defines hyperparameters, model & dataset settings.
- Data & Tokenization
– Data loader (inside train_lib, not explicit)
– videogvt/clip/simple_tokenizer.py, videogvt/interfaces/mm_vq.py
– Converts raw video frames to discrete tokens via 2D/3D VQGAN.
- Model Library
– VQ‐VAE encoders/decoders: enc_dec_2dcnn.py, enc_dec_3dcnn.py, vqvae.py
– Discriminators: stylegan_discriminator_*.py
– Video Transformer: transformer_lm.py, simplified_bert.py, t5_bert.py
- Training Pipeline
– Trainer Manager: videogvt/trainers/trainer_manager.py
– Trainers: maskgvt_trainer.py, vqgan_trainer.py, lmgvt_trainer.py
– Training Utilities: videogvt/train_lib (losses, mask schedule, sampling, train_state_manager)
- Evaluation & Tools
– Metrics code: compute_fvd.py, inception_score.py, image_quality_metrics.py
– Frechet Distance, Inception Score implementations in train_lib
- External Services & Storage
– Checkpoint storage (on‐disk or cloud buckets)
– Logging/visualization (e.g., TensorBoard via train_utils)
- Environment & Dependencies
– environment.yaml / requirements.txt
– JAX, Flax, NumPy, CUDA/CuDNN

##### Relationships & data flow
- Raw dataset → Tokenizer → Discrete video tokens
- Tokens → VQ‐VAE Encoder/Decoder (train or infer)
- Tokens → Transformer LM (train or infer)
- Trainer Manager uses Config → Instantiates the appropriate Trainer →
– Loads data, tokenizer, model(s)
– Orchestrates loss computation, backprop via train_lib
– Saves checkpoints, logs metrics
- Evaluation Tools read checkpoints + test data → compute FVD, IS, etc.
- CLI triggers all of the above, driven by user inputs and config.

##### Architectural patterns & design principles
- Modularity: clear separation of tokenization, modeling, training, and evaluation.
- Config‐driven: every experiment is defined by a config file.
- Trainer pattern: a generic Trainer Manager that dispatches to specific trainers.
- Test coverage: each util and trainer has corresponding *_test.py files.
- Dependency injection via train_lib utilities (e.g., task_registry, train_state_manager).

##### Relevant technologies & frameworks
- JAX/Flax for model implementation and training loops.
- Hydra‐style configs in pure Python (`.py` config files).
- NumPy, SciPy for metrics.
- Optional Colab integration via videogvt/interfaces/colab_utils.py.

##### Details to capture
- The three trainer types (VQGAN, MaskGVT, LM) and how Trainer Manager delegates to them.
- The train_lib layer handling state, loss, sampling, masking schedules.
- The dual‐stage modelling: first learn discrete tokens (VQ‐VAE), then learn sequence modelling (Transformer).
- Evaluation tools that hook into checkpoints to report FVD, IS, etc.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Config & CLI
    subgraph "Config & CLI" 
        direction TB
        Config["Configuration\n(videogvt/configs/*.py)"]:::config
        maskgvt_bair["maskgvt_bair_config.py"]:::config
        maskgvt_bair_multitask["maskgvt_bair_multitask_config.py"]:::config
        maskgvt_ucf101["maskgvt_ucf101_config.py"]:::config
        vqgan2d_ucf101["vqgan2d_ucf101_config.py"]:::config
        vqgan3d_bair["vqgan3d_bair_config.py"]:::config
        vqgan3d_ucf101["vqgan3d_ucf101_config.py"]:::config
        CLI["CLI Entry Point\nvideogvt/main.py"]:::config
    end

    %% Data & Tokenization
    subgraph "Data & Tokenization" 
        direction TB
        SimpleTokenizer["simple_tokenizer.py"]:::data
        MMVQ["mm_vq.py"]:::data
    end

    %% Modeling Components
    subgraph "Model Library" 
        direction TB
        subgraph "VQ-VAE Encoders/Decoders"
            direction TB
            Enc2D["enc_dec_2dcnn.py"]:::model
            Enc2plus1D["enc_dec_2plus1dcnn.py"]:::model
            Enc3D["enc_dec_3dcnn.py"]:::model
            VQVAE["vqvae.py"]:::model
        end
        subgraph "Discriminators"
            direction TB
            Disc2D["stylegan_discriminator_2d.py"]:::model
            Disc2plus1D["stylegan_discriminator_2plus1d.py"]:::model
            Disc3D["stylegan_discriminator_3d.py"]:::model
        end
        subgraph "Transformer & LM"
            direction TB
            TransformerLM["transformer_lm.py"]:::model
            SimplifiedBERT["simplified_bert.py"]:::model
            T5BERT["t5_bert.py"]:::model
        end
    end

    %% Training Pipeline
    subgraph "Training Pipeline" 
        direction TB
        TrainerManager["trainer_manager.py"]:::training
        subgraph "Trainers"
            direction TB
            VQGANTrainer["vqgan_trainer.py"]:::training
            MaskGVTTrainer["maskgvt_trainer.py"]:::training
            LMGVTTrainer["lmgvt_trainer.py"]:::training
        end
        subgraph "Training Utilities (train_lib)"
            direction TB
            StateManager["train_state_manager.py"]:::training
            TrainUtils["train_utils.py"]:::training
            TaskManager["task_manager.py"]:::training
            TaskRegistry["task_registry.py"]:::training
            Losses["losses.py"]:::training
            MaskSchedule["mask_schedule.py"]:::training
            MaskUtils["mask_utils.py"]:::training
            Sampling["sampling.py"]:::training
            ParallelDecode["parallel_decode.py"]:::training
            LMSampleDecode["lm_sample_decode.py"]:::training
            VQGANLib["vqgan_train_lib.py"]:::training
            MaskGVTLib["maskgvt_train_lib.py"]:::training
            FD["frechet_distance.py"]:::training
            IS["inception_score.py"]:::training
            IQM["image_quality_metrics.py"]:::training
            EvalUtils["eval_utils.py"]:::training
            CondUtils["cond_utils.py"]:::training
            PretrainedUtils["pretrained_model_utils.py"]:::training
            ResnetV1["resnet_v1.py"]:::training
        end
    end

    %% Evaluation & Tools
    subgraph "Evaluation & Tools"
        direction TB
        ComputeFVD["compute_fvd.py"]:::eval
    end

    %% External Dependencies & Storage
    subgraph "External"
        direction TB
        Checkpoint["Checkpoint Storage"]:::external
        Logging["Logging/Visualization\n(TensorBoard)"]:::external
        EnvYAML["environment.yaml"]:::external
        ReqTXT["requirements.txt"]:::external
    end

    %% Data Flow & Relationships
    CLI -->|uses config| Config
    Config --> maskgvt_bair
    Config --> maskgvt_bair_multitask
    Config --> maskgvt_ucf101
    Config --> vqgan2d_ucf101
    Config --> vqgan3d_bair
    Config --> vqgan3d_ucf101
    CLI --> TrainerManager
    TrainerManager --> VQGANTrainer
    TrainerManager --> MaskGVTTrainer
    TrainerManager --> LMGVTTrainer
    VQGANTrainer --> VQGANLib
    MaskGVTTrainer --> MaskGVTLib
    LMGVTTrainer --> Sampling
    TrainerManager --> StateManager
    TrainerManager --> TrainUtils
    TrainerManager --> TaskManager
    TrainerManager --> TaskRegistry
    TrainingPipeline ---|reads| SimpleTokenizer
    TrainingPipeline ---|reads| MMVQ
    SimpleTokenizer --> Enc2D
    SimpleTokenizer --> Enc2plus1D
    SimpleTokenizer --> Enc3D
    Enc2D --> VQVAE
    Enc2plus1D --> VQVAE
    Enc3D --> VQVAE
    VQVAE -->|tokens| TransformerLM
    TransformerLM -->|generates| Samples["Samples/Outputs"]:::model
    VQGANTrainer -->|saves| Checkpoint
    MaskGVTTrainer -->|saves| Checkpoint
    LMGVTTrainer -->|saves| Checkpoint
    Checkpoint --> ComputeFVD
    ComputeFVD -->|reports| Logging

    %% Click Events
    click CLI "https://github.com/google-research/magvit/blob/main/videogvt/main.py"
    click maskgvt_bair "https://github.com/google-research/magvit/blob/main/videogvt/configs/maskgvt_bair_config.py"
    click maskgvt_bair_multitask "https://github.com/google-research/magvit/blob/main/videogvt/configs/maskgvt_bair_multitask_config.py"
    click maskgvt_ucf101 "https://github.com/google-research/magvit/blob/main/videogvt/configs/maskgvt_ucf101_config.py"
    click vqgan2d_ucf101 "https://github.com/google-research/magvit/blob/main/videogvt/configs/vqgan2d_ucf101_config.py"
    click vqgan3d_bair "https://github.com/google-research/magvit/blob/main/videogvt/configs/vqgan3d_bair_config.py"
    click vqgan3d_ucf101 "https://github.com/google-research/magvit/blob/main/videogvt/configs/vqgan3d_ucf101_config.py"
    click SimpleTokenizer "https://github.com/google-research/magvit/blob/main/videogvt/clip/simple_tokenizer.py"
    click MMVQ "https://github.com/google-research/magvit/blob/main/videogvt/interfaces/mm_vq.py"
    click Enc2D "https://github.com/google-research/magvit/blob/main/videogvt/models/enc_dec_2dcnn.py"
    click Enc2plus1D "https://github.com/google-research/magvit/blob/main/videogvt/models/enc_dec_2plus1dcnn.py"
    click Enc3D "https://github.com/google-research/magvit/blob/main/videogvt/models/enc_dec_3dcnn.py"
    click VQVAE "https://github.com/google-research/magvit/blob/main/videogvt/models/vqvae.py"
    click Disc2D "https://github.com/google-research/magvit/blob/main/videogvt/models/stylegan_discriminator_2d.py"
    click Disc2plus1D "https://github.com/google-research/magvit/blob/main/videogvt/models/stylegan_discriminator_2plus1d.py"
    click Disc3D "https://github.com/google-research/magvit/blob/main/videogvt/models/stylegan_discriminator_3d.py"
    click TransformerLM "https://github.com/google-research/magvit/blob/main/videogvt/models/transformer_lm.py"
    click SimplifiedBERT "https://github.com/google-research/magvit/blob/main/videogvt/models/simplified_bert.py"
    click T5BERT "https://github.com/google-research/magvit/blob/main/videogvt/models/t5_bert.py"
    click TrainerManager "https://github.com/google-research/magvit/blob/main/videogvt/trainers/trainer_manager.py"
    click VQGANTrainer "https://github.com/google-research/magvit/blob/main/videogvt/trainers/vqgan_trainer.py"
    click MaskGVTTrainer "https://github.com/google-research/magvit/blob/main/videogvt/trainers/maskgvt_trainer.py"
    click LMGVTTrainer "https://github.com/google-research/magvit/blob/main/videogvt/trainers/lmgvt_trainer.py"
    click StateManager "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/train_state_manager.py"
    click TrainUtils "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/train_utils.py"
    click TaskManager "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/task_manager.py"
    click TaskRegistry "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/task_registry.py"
    click Losses "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/losses.py"
    click MaskSchedule "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/mask_schedule.py"
    click MaskUtils "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/mask_utils.py"
    click Sampling "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/sampling.py"
    click ParallelDecode "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/parallel_decode.py"
    click LMSampleDecode "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/lm_sample_decode.py"
    click VQGANLib "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/vqgan_train_lib.py"
    click MaskGVTLib "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/maskgvt_train_lib.py"
    click FD "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/frechet_distance.py"
    click IS "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/inception_score.py"
    click IQM "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/image_quality_metrics.py"
    click EvalUtils "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/eval_utils.py"
    click CondUtils "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/cond_utils.py"
    click PretrainedUtils "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/pretrained_model_utils.py"
    click ResnetV1 "https://github.com/google-research/magvit/blob/main/videogvt/train_lib/resnet_v1.py"
    click ComputeFVD "https://github.com/google-research/magvit/blob/main/videogvt/tools/compute_fvd.py"

    %% Styles
    classDef data fill:#ADD8E6
    classDef model fill:#90EE90
    classDef training fill:#FFA500
    classDef eval fill:#DDA0DD
    classDef config fill:#D3D3D3
    classDef external fill:#F0F0F0
```
