# valeoai/halton-maskgit architecture

> Architecture diagram of the GitHub repository valeoai/halton-maskgit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/valeoai/halton-maskgit
- Repository: https://github.com/valeoai/halton-maskgit

## Overview

##### Project Type and Purpose
This is a research-oriented, full-stack machine-learning codebase implementing “Halton Scheduler for Masked Generative Image Transformer (MaskGIT).” It provides end-to-end workflows for:
- Training a class-to-image generative model on ImageNet (via `Trainer/cls_trainer.py`).
- (Coming soon) Training a text-to-image model (`Trainer/txt_trainer.py`).
- Inference and sampling using custom schedulers (Halton, confidence).
- Evaluation metrics (FID, Inception score).
- Quick demos via Colab notebook and a Gradio app.

##### High-Level File Structure & Architectural Layers
- **Configuration Layer**
- `Config/` – YAML files (`base_cls2img.yaml`, `base_txt2img.yaml`) define hyperparameters, model paths, scheduler settings.
- `env.yaml` – Conda environment spec.

- **Data Layer**
- `Dataset/` – PyTorch `Dataset` and `DataLoader` implementations.
- Raw data resides externally (e.g., ImageNet folder passed to `extract_vq_features.py`).

- **Model Layer**
- `Network/` – Core PyTorch modules:
- `vq_model.py` (VQGAN encoder/decoder)
- `transformer.py` (MaskGIT transformer)
- `txt_transformer.py` (text → image transformer stub)
- `ema.py` (exponential moving average wrapper)

- **Sampling Layer**
- `Sampler/` – Two scheduling strategies:
- `halton_sampler.py` – Halton sequence scheduler
- `confidence_sampler.py` – Confidence-based scheduler

- **Training & Orchestration Layer**
- `Trainer/` – Training loop abstractions:
- `abstract_trainer.py` – Base trainer
- `cls_trainer.py`, `txt_trainer.py`
- `launch/` – Shell scripts (`run_cls_to_img.sh`, etc.) to spawn training/eval jobs.

- **Evaluation & Metrics Layer**
- `Metrics/` – Scripts for scoring and FID computation: `inception_metrics.py`, `sample_and_eval.py`, plus FID stat extraction.

- **Utilities & Demos**
- `Utils/` – Helpers for masking scheduler, general utils, visualization.
- `app.py` – Gradio web UI for real-time inference.
- `colab_demo.ipynb` – Colab demo notebook.

- **Storage & Results**
- `saved_networks/` – Pretrained VQGAN and MaskGIT weights.
- `results/` – Output images, logs.
- `statics/` – Sample images for README.

##### Key External Dependencies
- PyTorch
- HuggingFace Hub for model download (`hf_hub_download`)
- Conda environment
- (Optional) Gradio for web interface
- GPU cluster (e.g., IT4I Karolina)

A. **Configuration Manager**
- Input: YAML config files
- Outputs: parsed hyperparameters → Trainer & Sampler

B. **Data Ingestion**
- Image dataset (ImageNet) → `Dataset` and `DataLoader`

C. **Model Modules**
1. VQGAN Encoder/Decoder
2. Masked Transformer
3. EMA wrapper

D. **Trainer**
- Consumes data & model modules
- Produces checkpoints in `saved_networks/`

E. **Sampler**
- Halton or Confidence scheduler
- Generates image tokens from trained model

F. **Evaluation & Metrics**
- FID/Inception evaluation
- Uses outputs from Sampler

G. **Inference Interfaces**
1. Python CLI (`main.py`)
2. Gradio App (`app.py`)
3. Colab Notebook

H. **External Services**
- HuggingFace Model Hub
- GPU Compute Cluster

##### Interactions & Data Flows
1. **Offline Preprocessing**
Data → `extract_vq_features.py` → VQ codes → stored locally
[1] Config Manager
→ [2] Dataset & DataLoader
→ [3] Trainer ⟷ Model Modules
→ [4] Checkpoints in `saved_networks/`
3. **Inference Pipeline**
[5] Load config & checkpoint from HuggingFace/`saved_networks/`
→ [6] Sampler uses Model Modules
→ [7] `Utils.viz` → Generated images
Optionally: CLI, Gradio, or Notebook as front-ends.
4. **Evaluation Pipeline**
[8] `sample_and_eval.py`
↔ Model Modules + Sampler
→ Metrics (FID, IS)

##### Architectural Principles
- **Modularity & Separation of Concerns**: each layer (data, model, training, sampling, evaluation) is independently testable and replaceable.
- **Config-Driven Design**: hyperparameters and file paths are defined in YAML, enabling reproducibility.
- **Extensibility**: new schedulers (under `Sampler/`), new trainers (`Trainer/`), or new metrics (`Metrics/`) can be plugged in.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        CM1["base_cls2img.yaml"]:::configFile
        CM2["base_txt2img.yaml"]:::configFile
        ENV["env.yaml"]:::configFile
    end

    %% Data Layer
    subgraph "Data Layer"
        DS1["dataset.py"]:::moduleFile
        DS2["dataloader.py"]:::moduleFile
        EXTRACT["extract_vq_features.py"]:::moduleFile
        DATA_STORAGE["Raw ImageNet Data"]:::storage
    end

    %% Model Layer
    subgraph "Model Layer"
        VQ["vq_model.py"]:::moduleFile
        TRANS["transformer.py"]:::moduleFile
        TXT_TRANS["txt_transformer.py"]:::moduleFile
        EMA["ema.py"]:::moduleFile
    end

    %% Training & Orchestration Layer
    subgraph "Training Layer"
        AT["abstract_trainer.py"]:::moduleFile
        CT["cls_trainer.py"]:::moduleFile
        TT["txt_trainer.py"]:::moduleFile
        SH1["run_cls_to_img.sh"]:::moduleFile
        SH2["run_txt_to_img.sh"]:::moduleFile
        CHECKPOINTS["saved_networks"]:::storage
    end

    %% Sampling Layer
    subgraph "Sampling Layer"
        HALTON["halton_sampler.py"]:::moduleFile
        CONF["confidence_sampler.py"]:::moduleFile
    end

    %% Evaluation & Metrics Layer
    subgraph "Metrics Layer"
        INCP["inception_metrics.py"]:::moduleFile
        SAE["sample_and_eval.py"]:::moduleFile
        FID["extract_train_fid.py"]:::moduleFile
    end

    %% Inference Interfaces
    subgraph "Inference Interfaces"
        CLI["main.py"]:::interface
        APP["app.py"]:::interface
        COLAB["colab_demo.ipynb"]:::interface
    end

    %% External Services
    subgraph "External Services"
        HF["HuggingFace Hub"]:::external
        GPU["GPU Compute Cluster"]:::external
    end

    %% Storage & Results
    RESULTS["results"]:::storage
    STATICS["statics"]:::storage

    %% Flows
    CM1 -->|provides configs| DS1
    CM2 -->|provides configs| DS1
    ENV -->|environment| DS1

    DS1 -->|loads data| DS2
    DS2 -->|batches| CT
    DS2 -->|batches| TT
    EXTRACT -->|preprocess| VQ
    DATA_STORAGE -->|source| EXTRACT

    CM1 -->|provides configs| CT
    CM2 -->|provides configs| TT

    CT -->|trains| VQ
    CT -->|trains| TRANS
    CT -->|trains| EMA
    TT -->|trains| TRANS
    TT -->|trains| TXT_TRANS
    TT -->|trains| EMA

    CT -->|saves checkpoints| CHECKPOINTS
    TT -->|saves checkpoints| CHECKPOINTS
    SH1 -->|run| CT
    SH2 -->|run| TT

    CM1 -->|provides configs| HALTON
    CM1 -->|provides configs| CONF
    CHECKPOINTS -->|load| HALTON
    CHECKPOINTS -->|load| CONF
    VQ -->|encode/decode| HALTON
    TRANS -->|generate tokens| HALTON
    VQ -->|encode/decode| CONF
    TRANS -->|generate tokens| CONF

    HALTON -->|samples| RESULTS
    CONF -->|samples| RESULTS

    SAE -->|calls| VQ
    SAE -->|calls| TRANS
    SAE -->|calls| HALTON
    SAE -->|produces metrics| RESULTS
    INCP -->|computes FID/IS| RESULTS
    FID -->|extract stats| RESULTS

    HF -->|download checkpoints| CHECKPOINTS
    GPU -->|compute| CT
    GPU -->|compute| TT

    CHECKPOINTS -->|load| CLI
    CHECKPOINTS -->|load| APP
    CHECKPOINTS -->|load| COLAB
    HALTON -->|generate| CLI
    HALTON -->|generate| APP
    HALTON -->|generate| COLAB

    %% Click Events
    click CM1 "https://github.com/valeoai/halton-maskgit/blob/main/Config/base_cls2img.yaml"
    click CM2 "https://github.com/valeoai/halton-maskgit/blob/main/Config/base_txt2img.yaml"
    click ENV "https://github.com/valeoai/halton-maskgit/blob/main/env.yaml"
    click DS1 "https://github.com/valeoai/halton-maskgit/blob/main/Dataset/dataset.py"
    click DS2 "https://github.com/valeoai/halton-maskgit/blob/main/Dataset/dataloader.py"
    click EXTRACT "https://github.com/valeoai/halton-maskgit/blob/main/extract_vq_features.py"
    click VQ "https://github.com/valeoai/halton-maskgit/blob/main/Network/vq_model.py"
    click TRANS "https://github.com/valeoai/halton-maskgit/blob/main/Network/transformer.py"
    click TXT_TRANS "https://github.com/valeoai/halton-maskgit/blob/main/Network/txt_transformer.py"
    click EMA "https://github.com/valeoai/halton-maskgit/blob/main/Network/ema.py"
    click AT "https://github.com/valeoai/halton-maskgit/blob/main/Trainer/abstract_trainer.py"
    click CT "https://github.com/valeoai/halton-maskgit/blob/main/Trainer/cls_trainer.py"
    click TT "https://github.com/valeoai/halton-maskgit/blob/main/Trainer/txt_trainer.py"
    click SH1 "https://github.com/valeoai/halton-maskgit/blob/main/launch/run_cls_to_img.sh"
    click SH2 "https://github.com/valeoai/halton-maskgit/blob/main/launch/run_txt_to_img.sh"
    click HALTON "https://github.com/valeoai/halton-maskgit/blob/main/Sampler/halton_sampler.py"
    click CONF "https://github.com/valeoai/halton-maskgit/blob/main/Sampler/confidence_sampler.py"
    click INCP "https://github.com/valeoai/halton-maskgit/blob/main/Metrics/inception_metrics.py"
    click SAE "https://github.com/valeoai/halton-maskgit/blob/main/Metrics/sample_and_eval.py"
    click FID "https://github.com/valeoai/halton-maskgit/blob/main/Metrics/extract_train_fid.py"
    click CLI "https://github.com/valeoai/halton-maskgit/blob/main/main.py"
    click APP "https://github.com/valeoai/halton-maskgit/blob/main/app.py"
    click COLAB "https://github.com/valeoai/halton-maskgit/blob/main/colab_demo.ipynb"

    %% Styles
    classDef configFile fill:#D0E6FF,stroke:#0366D6,color:#0366D6;
    classDef moduleFile fill:#E8F5E9,stroke:#2E7D32,color:#2E7D32;
    classDef storage fill:#E8F8F5,stroke:#1B5E20,color:#1B5E20,shape:cylinder;
    classDef interface fill:#FFF3E0,stroke:#EF6C00,color:#E65100,shape:parallelogram;
    classDef external fill:#F5F5F5,stroke:#9E9E9E,color:#616161,shape:cloud;
```
