# lucidrains/autoregressive-diffusion-pytorch architecture

> Architecture diagram of the GitHub repository lucidrains/autoregressive-diffusion-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucidrains/autoregressive-diffusion-pytorch
- Repository: https://github.com/lucidrains/autoregressive-diffusion-pytorch

## Overview

1. **Repository purpose**
   - This is a small PyTorch package for **autoregressive generative modeling**, centered on the paper “Autoregressive Image Generation without Vector Quantization.”
   - It provides both the paper-style **autoregressive diffusion** path and an alternative **autoregressive flow / rectified flow** variant.

2. **Top-level package boundary**
   - The main deliverable is the Python package `autoregressive_diffusion_pytorch`.
   - `__init__.py` acts as the public API surface, exposing the user-facing model and training classes so downstream code can import from a single namespace.

3. **Core model subsystem**
   - `autoregressive_diffusion.py` contains the core **sequence autoregressive diffusion** model.
   - This is the model path used for non-image sequences and likely implements the training loss and sampling logic for token-by-token generation over a fixed maximum sequence length.

4. **Image-specialized model subsystem**
   - `autoregressive_flow.py` and/or the image-oriented wrapper classes provide the **image generation** path.
   - The README shows the package supports two image variants:
     - `ImageAutoregressiveDiffusion`
     - `ImageAutoregressiveFlow`
   - These likely adapt images into patch/token sequences and then delegate to the underlying autoregressive core.

5. **Image tokenization / sequence conversion**
   - Images are not generated as raw pixels directly; they are treated as **patch sequences**.
   - The critical architectural boundary is between:
     - image tensor input: `[batch, channels, height, width]`
     - sequence/token representation: `[batch, sequence_length, feature_dim]`
   - `image_size` and `patch_size` determine this conversion and therefore the sequence length.

6. **Training subsystem**
   - `image_trainer.py` contains the training orchestration for image experiments.
   - The README shows an `ImageTrainer` that wraps a model and a dataset, so the module likely owns the **dataset/dataloader/training loop** boundary rather than leaving that to the user.
   - This is the main place where optimization, batching, and iterative training are coordinated.

7. **Data source boundary**
   - The README references `ImageDataset('/path/to/your/images', image_size=...)`, implying the training path consumes a **local image directory** rather than a remote service or cloud datastore.
   - So the architecture’s external data boundary is simple: filesystem images in, batches into trainer, tensors into model.

8. **Inference / sampling flow**
   - Both sequence and image models expose a `.sample(batch_size=...)` path.
   - The runtime flow is:
     - model parameters loaded/initialized
     - optional conditioning implied by architecture, but not exposed in the README examples
     - model generates a full sequence
     - image wrapper reconstructs the sampled image tensor shape
   - Sampling is therefore a first-class subsystem, not just a training byproduct.

9. **Parallel algorithmic tracks**
   - The repo supports two closely related generative approaches:
     - **diffusion-based autoregression**
     - **flow-matching / rectified-flow-inspired autoregression**
   - This is an important architectural fork: the high-level API stays the same, but the underlying generative dynamics differ.

10. **Example / entry-point script**
   - `train_ar_flow_oxford.py` is a concrete experiment script, likely targeting the Oxford Flowers dataset referenced in the README image caption.
   - This file is not the library itself; it is an **application-level entry point** showing how to wire the package into a training run.

11. **Packaging and distribution**
   - `pyproject.toml` indicates standard Python packaging.
   - `.github/workflows/python-publish.yml` shows the project is configured for **automated package publishing**, so the repo is intended to be installed as a library rather than only run locally.

12. **Architectural diagram takeaway**
   - A useful diagram would show three layers:
     - **Public API/package layer** (`__init__.py`)
     - **Core generative models** (`autoregressive_diffusion.py`, `autoregressive_flow.py`)
     - **Image/training utilities** (`image_trainer.py`, dataset handling, `train_ar_flow_oxford.py`)
   - The main data path is: filesystem images or sequence tensors → patch/sequence representation → autoregressive model → loss during training or sampled outputs during inference.

Main entry points: [`autoregressive_diffusion_pytorch/image_trainer.py`](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/image_trainer.py) (Image Trainer), [`autoregressive_diffusion_pytorch/__init__.py`](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/__init__.py) (__init__.py), [`autoregressive_diffusion_pytorch/autoregressive_diffusion.py`](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/autoregressive_diffusion.py) (Autoregressive Diffusion), [`autoregressive_diffusion_pytorch/autoregressive_flow.py`](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/autoregressive_flow.py) (Autoregressive Flow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pkg["Python package"]
  node_init["__init__.py<br/>public API<br/>[__init__.py]"]
  node_ar_diff["Autoregressive Diffusion<br/>sequence model"]
  node_ar_flow["Autoregressive Flow<br/>flow model"]
end

subgraph group_image["Image pipeline"]
  node_trainer["Image Trainer<br/>training loop<br/>[image_trainer.py]"]
  node_image_ds[("Image Dataset<br/>filesystem data")]
  node_patch_seq["Patch Sequence<br/>image encoding"]
  node_img_ad["Image Diffusion<br/>image model"]
  node_img_af["Image Flow<br/>image model"]
  node_sample(("Sampling<br/>generation path"))
  node_flow_script["Oxford Trainer<br/>experiment script"]
end

subgraph group_release["Release tooling"]
  node_pyproject["pyproject.toml<br/>package config<br/>[pyproject.toml]"]
  node_publish["Publish workflow<br/>CI release<br/>[python-publish.yml]"]
end

node_init -->|"exports"| node_ar_diff
node_init -->|"exports"| node_ar_flow
node_trainer -->|"loads"| node_image_ds
node_trainer -->|"trains"| node_img_ad
node_trainer -->|"trains"| node_img_af
node_img_ad -->|"encodes"| node_patch_seq
node_img_af -->|"encodes"| node_patch_seq
node_patch_seq -->|"uses"| node_ar_diff
node_patch_seq -->|"uses"| node_ar_flow
node_ar_diff -->|"samples"| node_sample
node_ar_flow -->|"samples"| node_sample
node_sample -->|"reconstructs"| node_img_ad
node_sample -->|"reconstructs"| node_img_af
node_flow_script -->|"uses"| node_trainer
node_pyproject -->|"packages"| node_init
node_publish -->|"releases"| node_pyproject

click node_init "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/__init__.py"
click node_ar_diff "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/autoregressive_diffusion.py"
click node_ar_flow "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/autoregressive_flow.py"
click node_trainer "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/image_trainer.py"
click node_flow_script "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/train_ar_flow_oxford.py"
click node_pyproject "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/pyproject.toml"
click node_publish "https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/.github/workflows/python-publish.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_init,node_ar_diff,node_ar_flow toneBlue
class node_trainer,node_image_ds,node_patch_seq,node_img_ad,node_img_af,node_sample,node_flow_script toneAmber
class node_pyproject,node_publish toneMint
```

## Components

### Python package

Installable library surface and core model implementations.

- [__init__.py](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/__init__.py): public API · Exports the user-facing package namespace for downstream imports. · `autoregressive_diffusion_pytorch/__init__.py`
- [Autoregressive Diffusion](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/autoregressive_diffusion.py): sequence model · Core autoregressive diffusion model for token/sequence generation. · `autoregressive_diffusion_pytorch/autoregressive_diffusion.py`
- [Autoregressive Flow](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/autoregressive_flow.py): flow model · Alternative autoregressive flow / rectified-flow style generative model. · `autoregressive_diffusion_pytorch/autoregressive_flow.py`

### Image pipeline

Image-to-sequence adaptation, training, and sampling for image experiments.

- [Image Trainer](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/autoregressive_diffusion_pytorch/image_trainer.py): training loop · Coordinates image batching, optimization, and model training runs. · `autoregressive_diffusion_pytorch/image_trainer.py`
- **Image Dataset**: filesystem data · Loads local image directories and prepares them for training.
- **Patch Sequence**: image encoding · Converts images into patch/token sequences consumed by the models.
- **Image Diffusion**: image model · Image-oriented wrapper around the autoregressive diffusion path.
- **Image Flow**: image model · Image-oriented wrapper around the autoregressive flow path.
- **Sampling**: generation path · Produces full sequences and reconstructs sampled image outputs.
- [Oxford Trainer](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/train_ar_flow_oxford.py): experiment script · Concrete training entry point for an image flow experiment. · `train_ar_flow_oxford.py`

### Release tooling

Packaging and automated publish configuration.

- [pyproject.toml](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/pyproject.toml): package config · Defines build metadata and Python package distribution settings. · `pyproject.toml`
- [Publish workflow](https://github.com/lucidrains/autoregressive-diffusion-pytorch/blob/main/.github/workflows/python-publish.yml): CI release · Automated package publishing workflow on GitHub Actions. · `.github/workflows/python-publish.yml`

## Connections

- __init__.py → Autoregressive Diffusion: exports · Public API exposes the core diffusion model.
- __init__.py → Autoregressive Flow: exports · Public API exposes the flow variant.
- Image Trainer → Image Dataset: loads · Trainer consumes image data from local directories.
- Image Trainer → Image Diffusion: trains · Trainer can optimize the image diffusion model.
- Image Trainer → Image Flow: trains · Trainer can optimize the image flow model.
- Image Diffusion → Patch Sequence: encodes · Image diffusion operates on patch/token sequences.
- Image Flow → Patch Sequence: encodes · Image flow operates on patch/token sequences.
- Patch Sequence → Autoregressive Diffusion: uses · Image diffusion delegates sequence generation to the core diffusion model.
- Patch Sequence → Autoregressive Flow: uses · Image flow delegates sequence generation to the core flow model.
- Autoregressive Diffusion → Sampling: samples · The diffusion model supports autoregressive sampling.
- Autoregressive Flow → Sampling: samples · The flow model supports autoregressive sampling.
- Sampling → Image Diffusion: reconstructs · Image wrapper turns sampled sequences back into image tensors.
- Sampling → Image Flow: reconstructs · Image wrapper turns sampled sequences back into image tensors.
- Oxford Trainer → Image Trainer: uses · Experiment script wires the dataset and training loop together.
- pyproject.toml → __init__.py: packages · Package metadata builds and distributes the library namespace.
- Publish workflow → pyproject.toml: releases · CI workflow publishes the packaged distribution.
