How lucidrains/transfusion-pytorch works
Transfusion-Pytorch is a PyTorch library for jointly modeling interleaved text and one or more continuous modalities. The main workflow accepts mixed text/modality sequences, prepares modality-aware batches, trains a transformer with text-token and flow-matching objectives, and samples interleaved outputs. Sampling also supports batched prompts, classifier-free guidance, and optional modality encoders/decoders. The source excerpts are partial, so the graph focuses on documented behavior and visible module responsibilities; it omits uncertain internal call details and example training scripts.
Main entry points: transfusion.py Sampling · modality_processing.py Modality samples
Components 11
Model API
- Mixed sequences
- Transfusion
transfusion_pytorch/transfusion.py - Sampling
transfusion_pytorch/transfusion.py - Modality samples
transfusion_pytorch/modality_processing.py
Input Processing
- Processing strategies
transfusion_pytorch/modality_processing.py - Prepared batch
transfusion_pytorch/modality_processing.py
Model Runtime
- Transformer
transfusion_pytorch/transfusion.py - Text prediction
transfusion_pytorch/transfusion.py - Flow matching
transfusion_pytorch/transfusion.py - Modality codecs
transfusion_pytorch/transfusion.py
Other components
- Library caller
Connections 16
- Library caller → Transfusion: configures and calls
- Library caller → Mixed sequences: provides prompts
- Mixed sequences → Transfusion: input sequences
- Transfusion → Processing strategies: selects strategy
- Processing strategies → Prepared batch: assembles batch
- Prepared batch → Transformer: model input
- Transfusion → Transformer: runs forward
- Transformer → Text prediction: text predictions
- Transformer → Flow matching: modality predictions
- Transfusion → Modality codecs: encodes or decodes
- Library caller → Sampling: requests generation
- Sampling → Transformer: decodes tokens
- Sampling → Flow matching: integrates modality flow
- Sampling → Modality codecs: decodes modalities
- Sampling → Modality samples: returns samples
- Modality samples → Library caller: sample result