# foundationvision/infinity architecture

> Architecture diagram of the GitHub repository foundationvision/infinity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/foundationvision/infinity
- Repository: https://github.com/foundationvision/infinity

## Overview

##### Project Type & Purpose
- Infinity is an ML research codebase for “Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.”
- It provides:
- Data ingestion & preprocessing
- Visual tokenizer (bitwise VAE) and autoregressive transformer models
- Training pipeline with distributed & AMP support
- Evaluation modules (GenEval, ImageReward, HPSv2, FID, validation loss)
- Inference tools (scripts, notebooks, Docker)

##### Main Components
A. Data Layer
- data/infinity_toy_data (toy JSONL) & user-provided JSONL splits under data/
B. Model Layer (infinity/)
- infinity/dataset: JSONL iterable loader (dataset_t2i_iterable.py)
- infinity/models:
- bsq_vae (bitwise visual tokenizer)
- transformer architectures (basic, ema, flex_attn, t5 integration)
- bitwise self-correction, init, fused ops
- infinity/utils: distributed setup, AMP, argument parsing, I/O, saving/loading, LR control, wandb integration
C. Training Pipeline
- Entry points: train.py, trainer.py, scripts/train.sh
- Orchestration: torchrun, shell wrappers, cog.yaml (if using COG), DockerFile
- Checkpointing & logging: local_output folder & wandb
D. Evaluation Pipeline
- evaluation/gen_eval: datasets, default_runtime, model configs, prompts, scoring
- evaluation/hpsv2, image_reward, validation_loss
- scripts/eval.sh
E. Inference Pipeline
- predict.py, tools/*.py (comprehensive_infer, run_infinity, prompt_rewriter, interactive_infer notebooks)
- Docker container (DockerFile) for one-stop reproducibility
F. External Services & Dependencies
- HuggingFace: weights (VAE, Infinity models), flan-t5-xl tokenizer & model
- W&B for experiment tracking
- Compute cluster (torchrun, GPUs)

##### Relationships & Data Flows
1. Data Store → Dataset Iterator → Visual Tokenizer (VAE) → Bitwise Tokens
2. Tokens + Text Prompts (via flan-t5-xl) → Transformer Model → Bitwise Classifier → Loss
3. Trainer orchestrates forward/backward passes → Checkpoints & Logs
4. Checkpoints → Evaluation Modules (GenEval, HPSv2, ImageReward) → Metrics
5. Checkpoints + flan-t5-xl → Inference Tools → Generated Images
6. Docker wraps Inference & Reproduction scripts for end-to-end demo

##### Architectural Patterns & Design Principles
- Modular separation: data, model, train, eval, inference
- Plugin-style evaluation: independent subfolders under evaluation/
- Script wrappers for reproducibility and hyperparameterization
- Containerization for environment consistency

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DS[(Data Store)]:::data
        Labels[(Persistent Labels)]:::data
    end

    %% Compute & Model Layer
    subgraph "Compute/Model Layer"
        direction TB
        DI["Dataset Iterator"]:::compute
        VT["Visual Tokenizer (Bitwise VAE)"]:::compute
        subgraph "Transformer & Classifier"
            direction TB
            TM_basic["basic.py"]:::compute
            TM_bitcorr["bitwise_self_correction.py"]:::compute
            TM_infinity["infinity.py"]:::compute
            TM_flex["flex_attn.py"]:::compute
            TM_ema["ema.py"]:::compute
            TM_fused["fused_op.py"]:::compute
        end
        T5["External T5 Integration"]:::compute
    end

    %% Orchestration & Scripts Layer
    subgraph "Orchestration/Scripts Layer"
        direction TB
        Trainer["Trainer (train.py & trainer.py)"]:::orchestration
        TrainScript[/scripts/train.sh/]:::orchestration
        subgraph "Checkpoint & Logging"
            direction TB
            CP[(Checkpoint Storage)]:::orchestration
        end
        subgraph "Evaluation Modules"
            direction TB
            EvalGen["gen_eval/"]:::orchestration
            EvalHPS["hpsv2/eval_hpsv2.py"]:::orchestration
            EvalIR["image_reward/cal_imagereward.py"]:::orchestration
            EvalVal["validation_loss/validation_loss.py"]:::orchestration
            EvalScript[/scripts/eval.sh/]:::orchestration
        end
        subgraph "Inference Tools"
            direction TB
            Predict["predict.py"]:::orchestration
            CompInfer["tools/comprehensive_infer.py"]:::orchestration
            RunInf["tools/run_infinity.py"]:::orchestration
            PromptRW["tools/prompt_rewriter.py"]:::orchestration
            Notebook[/interactive_infer*.ipynb/]:::orchestration
        end
        Docker["Docker Container"]:::orchestration
    end

    %% External Services
    subgraph "External Services & Dependencies"
        direction TB
        HF["HuggingFace Weights & Tokenizer"]:::external
        WB["W&B Tracking"]:::external
        CC["Compute Cluster (GPUs)"]:::external
    end

    %% Connections
    DS --> DI
    Labels --> DI
    DI --> VT
    VT --> TM_basic
    VT --> TM_bitcorr
    VT --> TM_infinity
    VT --> TM_flex
    VT --> TM_ema
    VT --> TM_fused
    T5 --> TM_basic
    T5 --> TM_bitcorr
    T5 --> TM_infinity
    T5 --> TM_flex
    T5 --> TM_ema
    T5 --> TM_fused
    TM_basic --> Trainer
    TM_bitcorr --> Trainer
    TM_infinity --> Trainer
    TM_flex --> Trainer
    TM_ema --> Trainer
    TM_fused --> Trainer
    Trainer --> CP
    CP --> EvalGen
    CP --> EvalHPS
    CP --> EvalIR
    CP --> EvalVal
    CP --> Predict
    HF --> Predict
    Trainer --> WB
    Docker --> Predict
    Docker --> RunInf

    %% Click Events
    click DS "https://github.com/foundationvision/infinity/tree/main/data/infinity_toy_data/"
    click Labels "https://github.com/foundationvision/infinity/blob/main/data/labels/imagenet/val.txt"
    click DI "https://github.com/foundationvision/infinity/blob/main/infinity/dataset/dataset_t2i_iterable.py"
    click VT "https://github.com/foundationvision/infinity/blob/main/infinity/models/bsq_vae/vae.py"
    click TM_basic "https://github.com/foundationvision/infinity/blob/main/infinity/models/basic.py"
    click TM_bitcorr "https://github.com/foundationvision/infinity/blob/main/infinity/models/bitwise_self_correction.py"
    click TM_infinity "https://github.com/foundationvision/infinity/blob/main/infinity/models/infinity.py"
    click TM_flex "https://github.com/foundationvision/infinity/blob/main/infinity/models/flex_attn.py"
    click TM_ema "https://github.com/foundationvision/infinity/blob/main/infinity/models/ema.py"
    click TM_fused "https://github.com/foundationvision/infinity/blob/main/infinity/models/fused_op.py"
    click T5 "https://github.com/foundationvision/infinity/blob/main/infinity/models/t5.py"
    click Trainer "https://github.com/foundationvision/infinity/blob/main/trainer.py"
    click TrainScript "https://github.com/foundationvision/infinity/blob/main/scripts/train.sh"
    click CP "https://github.com/foundationvision/infinity/tree/main/local_output/"
    click EvalGen "https://github.com/foundationvision/infinity/tree/main/evaluation/gen_eval/"
    click EvalHPS "https://github.com/foundationvision/infinity/blob/main/evaluation/hpsv2/eval_hpsv2.py"
    click EvalIR "https://github.com/foundationvision/infinity/blob/main/evaluation/image_reward/cal_imagereward.py"
    click EvalVal "https://github.com/foundationvision/infinity/blob/main/evaluation/validation_loss/validation_loss.py"
    click EvalScript "https://github.com/foundationvision/infinity/blob/main/scripts/eval.sh"
    click Predict "https://github.com/foundationvision/infinity/blob/main/predict.py"
    click CompInfer "https://github.com/foundationvision/infinity/blob/main/tools/comprehensive_infer.py"
    click RunInf "https://github.com/foundationvision/infinity/blob/main/tools/run_infinity.py"
    click PromptRW "https://github.com/foundationvision/infinity/blob/main/tools/prompt_rewriter.py"
    click Notebook "https://github.com/foundationvision/infinity/blob/main/tools/interactive_infer.ipynb"
    click Docker "https://github.com/foundationvision/infinity/tree/main/DockerFile"
    click Docker "https://github.com/foundationvision/infinity/blob/main/cog.yaml"

    %% Styles
    classDef data fill:#cce5ff,stroke:#3399ff,color:#000;
    classDef compute fill:#d4edda,stroke:#28a745,color:#000;
    classDef orchestration fill:#ffe5cc,stroke:#ff9933,color:#000;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#000;
```
