How google-research/magvit works
Main components
• Command‐Line Interface (CLI) / Entry Point
– videogvt/main.py
– Reads CLI args, loads a config, dispatches to training, evaluation, or sampling.
• Configuration Layer
– videogvt/configs/*.py
– Defines hyperparameters, model & dataset settings.
• Data & Tokenization
– Data loader (inside train_lib, not explicit)
– videogvt/clip/simple_tokenizer.py, videogvt/interfaces/mm_vq.py
– Converts raw video frames to discrete tokens via 2D/3D VQGAN.
• Model Library
– VQ‐VAE encoders/decoders: enc_dec_2dcnn.py, enc_dec_3dcnn.py, vqvae.py
– Discriminators: stylegan_discriminator_*.py
– Video Transformer: transformer_lm.py, simplified_bert.py, t5_bert.py
• Training Pipeline
– Trainer Manager: videogvt/trainers/trainer_manager.py
– Trainers: maskgvt_trainer.py, vqgan_trainer.py, lmgvt_trainer.py
– Training Utilities: videogvt/train_lib (losses, mask schedule, sampling, train_state_manager)
• Evaluation & Tools
– Metrics code: compute_fvd.py, inception_score.py, image_quality_metrics.py
– Frechet Distance, Inception Score implementations in train_lib
• External Services & Storage
– Checkpoint storage (on‐disk or cloud buckets)
– Logging/visualization (e.g., TensorBoard via train_utils)
• Environment & Dependencies
– environment.yaml / requirements.txt
– JAX, Flax, NumPy, CUDA/CuDNN
Relationships & data flow
• Raw dataset → Tokenizer → Discrete video tokens
• Tokens → VQ‐VAE Encoder/Decoder (train or infer)
• Tokens → Transformer LM (train or infer)
• Trainer Manager uses Config → Instantiates the appropriate Trainer →
– Loads data, tokenizer, model(s)
– Orchestrates loss computation, backprop via train_lib
– Saves checkpoints, logs metrics
• Evaluation Tools read checkpoints + test data → compute FVD, IS, etc.
• CLI triggers all of the above, driven by user inputs and config.
Architectural patterns & design principles
• Modularity: clear separation of tokenization, modeling, training, and evaluation.
• Config‐driven: every experiment is defined by a config file.
• Trainer pattern: a generic Trainer Manager that dispatches to specific trainers.
• Test coverage: each util and trainer has corresponding *_test.py files.
• Dependency injection via train_lib utilities (e.g., task_registry, train_state_manager).
Relevant technologies & frameworks
• JAX/Flax for model implementation and training loops.
• Hydra‐style configs in pure Python (.py config files).
• NumPy, SciPy for metrics.
• Optional Colab integration via videogvt/interfaces/colab_utils.py.
Details to capture
• The three trainer types (VQGAN, MaskGVT, LM) and how Trainer Manager delegates to them.
• The train_lib layer handling state, loss, sampling, masking schedules.
• The dual‐stage modelling: first learn discrete tokens (VQ‐VAE), then learn sequence modelling (Transformer).
• Evaluation tools that hook into checkpoints to report FVD, IS, etc.