How valeoai/halton-maskgit works
Project Type and Purpose
This is a research-oriented, full-stack machine-learning codebase implementing “Halton Scheduler for Masked Generative Image Transformer (MaskGIT).” It provides end-to-end workflows for:
• Training a class-to-image generative model on ImageNet (via Trainer/cls_trainer.py).
• (Coming soon) Training a text-to-image model (Trainer/txt_trainer.py).
• Inference and sampling using custom schedulers (Halton, confidence).
• Evaluation metrics (FID, Inception score).
• Quick demos via Colab notebook and a Gradio app.
High-Level File Structure & Architectural Layers
• Configuration Layer
• Config/ – YAML files (base_cls2img.yaml, base_txt2img.yaml) define hyperparameters, model paths, scheduler settings.
• env.yaml – Conda environment spec.
• Data Layer
• Dataset/ – PyTorch Dataset and DataLoader implementations.
• Raw data resides externally (e.g., ImageNet folder passed to extract_vq_features.py).
• Model Layer
• Network/ – Core PyTorch modules:
• vq_model.py (VQGAN encoder/decoder)
• transformer.py (MaskGIT transformer)
• txt_transformer.py (text → image transformer stub)
• ema.py (exponential moving average wrapper)
• Sampling Layer
• Sampler/ – Two scheduling strategies:
• halton_sampler.py – Halton sequence scheduler
• confidence_sampler.py – Confidence-based scheduler
• Training & Orchestration Layer
• Trainer/ – Training loop abstractions:
• abstract_trainer.py – Base trainer
• cls_trainer.py, txt_trainer.py
• launch/ – Shell scripts (run_cls_to_img.sh, etc.) to spawn training/eval jobs.
• Evaluation & Metrics Layer
• Metrics/ – Scripts for scoring and FID computation: inception_metrics.py, sample_and_eval.py, plus FID stat extraction.
• Utilities & Demos
• Utils/ – Helpers for masking scheduler, general utils, visualization.
• app.py – Gradio web UI for real-time inference.
• colab_demo.ipynb – Colab demo notebook.
• Storage & Results
• saved_networks/ – Pretrained VQGAN and MaskGIT weights.
• results/ – Output images, logs.
• statics/ – Sample images for README.
Key External Dependencies
• PyTorch
• HuggingFace Hub for model download (hf_hub_download)
• Conda environment
• (Optional) Gradio for web interface
• GPU cluster (e.g., IT4I Karolina)
A. Configuration Manager
• Input: YAML config files
• Outputs: parsed hyperparameters → Trainer & Sampler
B. Data Ingestion
• Image dataset (ImageNet) → Dataset and DataLoader
C. Model Modules
1. VQGAN Encoder/Decoder
2. Masked Transformer
3. EMA wrapper
D. Trainer
• Consumes data & model modules
• Produces checkpoints in saved_networks/
E. Sampler
• Halton or Confidence scheduler
• Generates image tokens from trained model
F. Evaluation & Metrics
• FID/Inception evaluation
• Uses outputs from Sampler
G. Inference Interfaces
1. Python CLI (main.py)
2. Gradio App (app.py)
3. Colab Notebook
H. External Services
• HuggingFace Model Hub
• GPU Compute Cluster
Interactions & Data Flows
1. Offline Preprocessing
Data → extract_vq_features.py → VQ codes → stored locally
[1] Config Manager
→ [2] Dataset & DataLoader
→ [3] Trainer ⟷ Model Modules
→ [4] Checkpoints in saved_networks/
3. Inference Pipeline
[5] Load config & checkpoint from HuggingFace/saved_networks/
→ [6] Sampler uses Model Modules
→ [7] Utils.viz → Generated images
Optionally: CLI, Gradio, or Notebook as front-ends.
4. Evaluation Pipeline
[8] sample_and_eval.py
↔ Model Modules + Sampler
→ Metrics (FID, IS)
Architectural Principles
• Modularity & Separation of Concerns: each layer (data, model, training, sampling, evaluation) is independently testable and replaceable.
• Config-Driven Design: hyperparameters and file paths are defined in YAML, enabling reproducibility.
• Extensibility: new schedulers (under Sampler/), new trainers (Trainer/), or new metrics (Metrics/) can be plugged in.