# facebookresearch/mixture-of-transformers architecture

> Architecture diagram of the GitHub repository facebookresearch/mixture-of-transformers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/mixture-of-transformers
- Repository: https://github.com/facebookresearch/mixture-of-transformers

## Overview

This repository is a basic PyTorch implementation of Mixture-of-Transformers components, intended to be integrated into a host transformer. Its runtime scope is modality-specific feed-forward and attention layers, plus shared merging and normalization utilities; the tree contains no full model, data pipeline, or generation entry point. The README also describes broader Chameleon and Transfusion settings, but their implementations are not present in the supplied tree.

Main entry points: [`src/simple_ModalityUntiedAttention.py`](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/simple_ModalityUntiedAttention.py) (Modality attention), [`src/simple_ModalityUntiedFeedForward.py`](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/simple_ModalityUntiedFeedForward.py) (Modality FFN), [`src/utils.py`](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/utils.py) (Merge modalities)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_layers["MoT layers"]
  node_ffn["Modality FFN"]
  node_attention["Modality attention"]
end

subgraph group_shared["Shared utilities"]
  node_merge["Merge modalities<br/>[utils.py]"]
  node_norm["RMS normalization<br/>[utils.py]"]
end

node_host(("Host transformer"))
node_mha["PyTorch attention"]

node_host -->|"supplies tokens/masks"| node_ffn
node_host -->|"supplies tokens/masks"| node_attention
node_ffn -->|"normalizes experts"| node_norm
node_ffn -->|"merges outputs"| node_merge
node_attention -->|"normalizes experts"| node_norm
node_attention -->|"merges QKV/output"| node_merge
node_attention -->|"computes attention"| node_mha
node_ffn -->|"returns output"| node_host
node_attention -->|"returns output/weights"| node_host

click node_ffn "https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/simple_ModalityUntiedFeedForward.py"
click node_attention "https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/simple_ModalityUntiedAttention.py"
click node_merge "https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/utils.py"
click node_norm "https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ffn,node_attention toneBlue
class node_merge,node_norm toneAmber
class node_host,node_mha toneIndigo
```

## Components

### MoT layers

- [Modality FFN](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/simple_ModalityUntiedFeedForward.py): `src/simple_ModalityUntiedFeedForward.py`
- [Modality attention](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/simple_ModalityUntiedAttention.py): `src/simple_ModalityUntiedAttention.py`

### Shared utilities

- [Merge modalities](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/utils.py): `src/utils.py`
- [RMS normalization](https://github.com/facebookresearch/mixture-of-transformers/blob/main/src/utils.py): `src/utils.py`

### Other components

- **Host transformer**
- **PyTorch attention**

## Connections

- Host transformer → Modality FFN: supplies tokens/masks
- Host transformer → Modality attention: supplies tokens/masks
- Modality FFN → RMS normalization: normalizes experts
- Modality FFN → Merge modalities: merges outputs
- Modality attention → RMS normalization: normalizes experts
- Modality attention → Merge modalities: merges QKV/output
- Modality attention → PyTorch attention: computes attention
- Modality FFN → Host transformer: returns output
- Modality attention → Host transformer: returns output/weights