How facebookresearch/mixture-of-transformers works
This repository is a basic PyTorch implementation of Mixture-of-Transformers components, intended to be integrated into a host transformer. Its runtime scope is modality-specific feed-forward and attention layers, plus shared merging and normalization utilities; the tree contains no full model, data pipeline, or generation entry point. The README also describes broader Chameleon and Transfusion settings, but their implementations are not present in the supplied tree.
Main entry points: simple_ModalityUntiedAttention.py Modality attention · simple_ModalityUntiedFeedForward.py Modality FFN · utils.py Merge modalities
Components 6
MoT layers
- Modality FFN
src/simple_ModalityUntiedFeedForward.py - Modality attention
src/simple_ModalityUntiedAttention.py
Shared utilities
- Merge modalities
src/utils.py - RMS normalization
src/utils.py
Other components
- Host transformer
- PyTorch attention
Connections 9
- Host transformer → Modality FFN: supplies tokens/masks
- Host transformer → Modality attention: supplies tokens/masks
- Modality FFN → RMS normalization: normalizes experts
- Modality FFN → Merge modalities: merges outputs
- Modality attention → RMS normalization: normalizes experts
- Modality attention → Merge modalities: merges QKV/output
- Modality attention → PyTorch attention: computes attention
- Modality FFN → Host transformer: returns output
- Modality attention → Host transformer: returns output/weights