# google-research/maskgit architecture

> Architecture diagram of the GitHub repository google-research/maskgit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-research/maskgit
- Repository: https://github.com/google-research/maskgit

## Overview

##### Project type and purpose
- MaskGIT is a research-oriented JAX implementation of a masked generative image transformer.
- Its core function is to take images (or class labels) → tokenize via a VQGAN tokenizer → apply a bidirectional Transformer with a masking schedule → reconstruct or generate images.

##### File structure
Top level:
- MaskGIT_demo.ipynb – interactive demo/inference notebook
- maskgit/ – main Python package
- imgs/ – sample images for README
- requirements.txt, LICENSE, CONTRIBUTING.md

Inside maskgit/:
- configs/ – all experiment and model hyperparameters
– base_config.py
– maskgit_class_cond_config.py
– vqgan_config.py
- nets/ – neural network definitions
– vqgan_tokenizer.py (encoder/decoder)
– maskgit_transformer.py (Transformer decoder)
– discriminator.py (auxiliary GAN loss)
– layers.py (building blocks)
- libml/ – ML utilities
– losses.py (reconstruction, adversarial)
– mask_schedule.py (masking strategy)
– parallel_decode.py (iterative refinement)
- inference.py – orchestration of inference pipeline
- utils.py – shared helper functions

##### Extract architecture insights from the README
- Training (not yet implemented in repo but implied) will use configs + nets + libml to learn to predict masked tokens.
- Inference uses pretrained checkpoints to generate or edit images, as demonstrated in the demo notebook.
- External dependencies: JAX/Flax, dataset storage (e.g., ImageNet), Google Cloud Storage (for checkpoints).

##### Main components and their relationships
a. Configuration Manager
- Loads experiment parameters (model size, mask schedule, optimizer settings).
b. Data Pipeline (Training)
- Dataset loader → images → VQGAN tokenizer → discrete tokens → create masked tokens via mask_schedule → feed to Transformer → compute losses (reconstruction + optional GAN) via libml/losses → optimizer step.
c. Model Components
1. VQGAN Tokenizer
– Encoder: image → tokens
– Decoder: tokens → image
2. MaskGIT Transformer
– Bidirectional decoder predicting masked tokens.
3. Discriminator (optional GAN loss)
d. Inference Pipeline
- inference.py orchestrates:
1. Load config & pretrained checkpoints
2. Initialize VQGAN tokenizer + Transformer weights
3. Create initial full mask (all tokens masked)
4. Run parallel_decode: iteratively unmask/refine tokens
5. Decode tokens back to image via VQGAN decoder
e. Utilities
- parallel_decode, mask_schedule, utils for data sharding, logging, checkpoint I/O.

##### Architectural patterns and design principles
- Modularity: configs ↔ nets ↔ libml ↔ inference
- Separation of concerns: model definition vs. training utilities vs. inference orchestration
- Functional pipelines: data in → transform → model → losses/inference → output

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data and Services
    DataSource["ImageNet Dataset"]:::external
    Checkpoints["Checkpoint Storage"]:::external
    JAX["JAX/Flax"]:::external

    %% Configuration & Orchestration (Blue)
    subgraph "Configuration & Orchestration"
        direction TB
        Config["Configuration Manager\n(maskgit/configs/)"]:::config
        DemoNotebook["MaskGIT_demo.ipynb"]:::config
        InferencePy["inference.py"]:::config
        Utils["utils.py"]:::util
    end

    %% Model Definitions (Green)
    subgraph "Model Definitions"
        direction TB
        VQGANTokenizer["VQGAN Tokenizer\n(Encoder/Decoder)"]:::model
        MaskGITTransformer["MaskGIT Transformer"]:::model
        Discriminator["Discriminator\n(Aux GAN Loss)"]:::model
        Layers["Model Building Blocks"]:::model
    end

    %% Utilities & ML Functions (Orange)
    subgraph "Utilities & ML Functions"
        direction TB
        Losses["losses.py\n(Reconstruction & Adversarial)"]:::util
        MaskSchedule["mask_schedule.py"]:::util
        ParallelDecode["parallel_decode.py\n(Iterative Refinement)"]:::util
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        DataSource --> VQGANTokenizer
        VQGANTokenizer --> MaskSchedule
        MaskSchedule --> MaskedTokens["Masked Tokens"]
        MaskedTokens --> MaskGITTransformer
        MaskGITTransformer --> PredictedTokens["Predicted Tokens"]
        DataSource --> RealTokens["Real Tokens"]
        PredictedTokens --> Losses
        RealTokens --> Losses
        Losses --> Optimizer["Optimizer"]
        Config --> MaskSchedule
        Config --> MaskGITTransformer
        Config --> Losses
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        DemoNotebook --> InferencePy
        Checkpoints --> InferencePy
        Config --> InferencePy
        InferencePy --> VQGANTokenizer
        InferencePy --> MaskGITTransformer
        InferencePy --> MaskSchedule
        MaskSchedule --> InitialMask["Initial Full Mask"]
        InitialMask --> ParallelDecode
        ParallelDecode --> PredTokensInf["Predicted Tokens"]
        PredTokensInf --> VQGANDecoder["VQGAN Decoder\n(Generator)"]:::model
        VQGANDecoder --> OutputImage["Output Image"]
    end

    %% External dependencies
    JAX -.-> VQGANTokenizer
    JAX -.-> MaskGITTransformer
    JAX -.-> InferencePy

    %% Click Events
    click Config "https://github.com/google-research/maskgit/tree/main/maskgit/configs/"
    click VQGANTokenizer "https://github.com/google-research/maskgit/blob/main/maskgit/nets/vqgan_tokenizer.py"
    click MaskGITTransformer "https://github.com/google-research/maskgit/blob/main/maskgit/nets/maskgit_transformer.py"
    click Discriminator "https://github.com/google-research/maskgit/blob/main/maskgit/nets/discriminator.py"
    click Layers "https://github.com/google-research/maskgit/blob/main/maskgit/nets/layers.py"
    click Losses "https://github.com/google-research/maskgit/blob/main/maskgit/libml/losses.py"
    click MaskSchedule "https://github.com/google-research/maskgit/blob/main/maskgit/libml/mask_schedule.py"
    click ParallelDecode "https://github.com/google-research/maskgit/blob/main/maskgit/libml/parallel_decode.py"
    click InferencePy "https://github.com/google-research/maskgit/blob/main/maskgit/inference.py"
    click DemoNotebook "https://github.com/google-research/maskgit/blob/main/MaskGIT_demo.ipynb"
    click Utils "https://github.com/google-research/maskgit/blob/main/maskgit/utils.py"

    %% Styles
    classDef config fill:#cce5ff,stroke:#005cbf,color:#005cbf;
    classDef model fill:#d4edda,stroke:#155724,color:#155724;
    classDef util fill:#fff3cd,stroke:#856404,color:#856404;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
