# lucidrains/transfusion-pytorch architecture

> Architecture diagram of the GitHub repository lucidrains/transfusion-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucidrains/transfusion-pytorch
- Repository: https://github.com/lucidrains/transfusion-pytorch

## Overview

Transfusion-Pytorch is a PyTorch library for jointly modeling interleaved text and one or more continuous modalities. The main workflow accepts mixed text/modality sequences, prepares modality-aware batches, trains a transformer with text-token and flow-matching objectives, and samples interleaved outputs. Sampling also supports batched prompts, classifier-free guidance, and optional modality encoders/decoders. The source excerpts are partial, so the graph focuses on documented behavior and visible module responsibilities; it omits uncertain internal call details and example training scripts.

Main entry points: [`transfusion_pytorch/transfusion.py`](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py) (Sampling), [`transfusion_pytorch/modality_processing.py`](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py) (Modality samples)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Model API"]
  node_mixed_data["Mixed sequences"]
  node_transfusion["Transfusion<br/>[transfusion.py]"]
  node_sampling["Sampling<br/>[transfusion.py]"]
  node_sample_output["Modality samples"]
end

subgraph group_processing["Input Processing"]
  node_processing_strategy["Processing strategies"]
  node_prepared_batch["Prepared batch"]
end

subgraph group_runtime["Model Runtime"]
  node_transformer["Transformer<br/>[transfusion.py]"]
  node_text_objective["Text prediction<br/>[transfusion.py]"]
  node_flow_objective["Flow matching<br/>[transfusion.py]"]
  node_codec["Modality codecs<br/>[transfusion.py]"]
end

node_developer(("Library caller"))

node_developer -->|"configures and calls"| node_transfusion
node_developer -->|"provides prompts"| node_mixed_data
node_mixed_data -->|"input sequences"| node_transfusion
node_transfusion -->|"selects strategy"| node_processing_strategy
node_processing_strategy -->|"assembles batch"| node_prepared_batch
node_prepared_batch -->|"model input"| node_transformer
node_transfusion -->|"runs forward"| node_transformer
node_transformer -->|"text predictions"| node_text_objective
node_transformer -->|"modality predictions"| node_flow_objective
node_transfusion -.->|"encodes or decodes"| node_codec
node_developer -->|"requests generation"| node_sampling
node_sampling -->|"decodes tokens"| node_transformer
node_sampling -->|"integrates modality flow"| node_flow_objective
node_sampling -.->|"decodes modalities"| node_codec
node_sampling -->|"returns samples"| node_sample_output
node_sample_output -->|"sample result"| node_developer

click node_transfusion "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py"
click node_processing_strategy "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py"
click node_prepared_batch "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py"
click node_transformer "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py"
click node_text_objective "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py"
click node_flow_objective "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py"
click node_codec "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py"
click node_sampling "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py"
click node_sample_output "https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mixed_data,node_transfusion,node_sampling,node_sample_output toneBlue
class node_processing_strategy,node_prepared_batch toneAmber
class node_transformer,node_text_objective,node_flow_objective,node_codec toneMint
class node_developer toneIndigo
```

## Components

### Model API

- **Mixed sequences**
- [Transfusion](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py): `transfusion_pytorch/transfusion.py`
- [Sampling](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py): `transfusion_pytorch/transfusion.py`
- [Modality samples](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py): `transfusion_pytorch/modality_processing.py`

### Input Processing

- [Processing strategies](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py): `transfusion_pytorch/modality_processing.py`
- [Prepared batch](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/modality_processing.py): `transfusion_pytorch/modality_processing.py`

### Model Runtime

- [Transformer](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py): `transfusion_pytorch/transfusion.py`
- [Text prediction](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py): `transfusion_pytorch/transfusion.py`
- [Flow matching](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py): `transfusion_pytorch/transfusion.py`
- [Modality codecs](https://github.com/lucidrains/transfusion-pytorch/blob/main/transfusion_pytorch/transfusion.py): `transfusion_pytorch/transfusion.py`

### Other components

- **Library caller**

## Connections

- Library caller → Transfusion: configures and calls
- Library caller → Mixed sequences: provides prompts
- Mixed sequences → Transfusion: input sequences
- Transfusion → Processing strategies: selects strategy
- Processing strategies → Prepared batch: assembles batch
- Prepared batch → Transformer: model input
- Transfusion → Transformer: runs forward
- Transformer → Text prediction: text predictions
- Transformer → Flow matching: modality predictions
- Transfusion → Modality codecs: encodes or decodes
- Library caller → Sampling: requests generation
- Sampling → Transformer: decodes tokens
- Sampling → Flow matching: integrates modality flow
- Sampling → Modality codecs: decodes modalities
- Sampling → Modality samples: returns samples
- Modality samples → Library caller: sample result