How foundationvision/infinity works
Project Type & Purpose
• Infinity is an ML research codebase for “Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.”
• It provides:
• Data ingestion & preprocessing
• Visual tokenizer (bitwise VAE) and autoregressive transformer models
• Training pipeline with distributed & AMP support
• Evaluation modules (GenEval, ImageReward, HPSv2, FID, validation loss)
• Inference tools (scripts, notebooks, Docker)
Main Components
A. Data Layer
• data/infinity_toy_data (toy JSONL) & user-provided JSONL splits under data/
B. Model Layer (infinity/)
• infinity/dataset: JSONL iterable loader (dataset_t2i_iterable.py)
• infinity/models:
• bsq_vae (bitwise visual tokenizer)
• transformer architectures (basic, ema, flex_attn, t5 integration)
• bitwise self-correction, init, fused ops
• infinity/utils: distributed setup, AMP, argument parsing, I/O, saving/loading, LR control, wandb integration
C. Training Pipeline
• Entry points: train.py, trainer.py, scripts/train.sh
• Orchestration: torchrun, shell wrappers, cog.yaml (if using COG), DockerFile
• Checkpointing & logging: local_output folder & wandb
D. Evaluation Pipeline
• evaluation/gen_eval: datasets, default_runtime, model configs, prompts, scoring
• evaluation/hpsv2, image_reward, validation_loss
• scripts/eval.sh
E. Inference Pipeline
• predict.py, tools/*.py (comprehensive_infer, run_infinity, prompt_rewriter, interactive_infer notebooks)
• Docker container (DockerFile) for one-stop reproducibility
F. External Services & Dependencies
• HuggingFace: weights (VAE, Infinity models), flan-t5-xl tokenizer & model
• W&B for experiment tracking
• Compute cluster (torchrun, GPUs)
Relationships & Data Flows
1. Data Store → Dataset Iterator → Visual Tokenizer (VAE) → Bitwise Tokens
2. Tokens + Text Prompts (via flan-t5-xl) → Transformer Model → Bitwise Classifier → Loss
3. Trainer orchestrates forward/backward passes → Checkpoints & Logs
4. Checkpoints → Evaluation Modules (GenEval, HPSv2, ImageReward) → Metrics
5. Checkpoints + flan-t5-xl → Inference Tools → Generated Images
6. Docker wraps Inference & Reproduction scripts for end-to-end demo
Architectural Patterns & Design Principles
• Modular separation: data, model, train, eval, inference
• Plugin-style evaluation: independent subfolders under evaluation/
• Script wrappers for reproducibility and hyperparameterization
• Containerization for environment consistency