# microsoft/unilm architecture

> Architecture diagram of the GitHub repository microsoft/unilm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/unilm
- Repository: https://github.com/microsoft/unilm

## Overview

This repository is a research portfolio, not one integrated application: its README presents separate foundation-model architectures, training toolkits, and task-specific applications. The sampled code supports a few local workflows, including BEiT image dataset construction and Fairseq multilingual dataset loading; there is no evidence for a single user-to-result path across the repository. Nodes also retain the README’s major product branches. Connections are limited to relationships visible in sampled excerpts or SOURCE INDEX references; unsampled subsystems are represented without inferred internal wiring.

Main entry points: [`edgelm/fairseq/data/multilingual/multilingual_data_manager.py`](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/multilingual/multilingual_data_manager.py) (Multilingual manager), [`edgelm/fairseq/data/data_utils.py`](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/data_utils.py) (Dataset utilities), [`beit/datasets.py`](https://github.com/microsoft/unilm/blob/master/beit/datasets.py) (BEiT data), [`edgelm/fairseq/data/indexed_dataset.py`](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/indexed_dataset.py) (Indexed datasets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_arch["Model architectures"]
  node_diff_transformer["Diff Transformer"]
  node_yoco["YOCO<br/>[yoco.py]"]
  node_resa["ReSA LLM<br/>[model.py]"]
end

subgraph group_training["Training and data"]
  node_beit["BEiT"]
  node_beit_data["BEiT data<br/>[datasets.py]"]
  node_beit_transforms["Image transforms<br/>[transforms.py]"]
  node_beit_masks["Mask generation"]
  node_beit2["BEiT-2<br/>[modeling_vqkd.py]"]
  node_beit3["BEiT-3"]
  node_multilingual["Multilingual manager"]
  node_data_utils["Dataset utilities<br/>[data_utils.py]"]
  node_indexed_data[("Indexed datasets<br/>[indexed_dataset.py]")]
  node_sampled_multi["Sampled datasets"]
  node_file_io["Path manager<br/>[file_io.py]"]
  node_adalm["AdaLM adaptation<br/>[run_classifier.py]"]
end

subgraph group_applications["Applications"]
  node_latent_inference["LatentLM inference<br/>[inference_speed.py]"]
  node_latent_solver["Diffusion solver<br/>[dpm_solver.py]"]
  node_latent_models["LatentLM models<br/>[DiT.py]"]
  node_textdiffuser["TextDiffuser<br/>[inference.py]"]
  node_textdiffuser2["TextDiffuser-2"]
  node_kosmos25["Kosmos-2.5<br/>[inference.py]"]
  node_trocr["TrOCR<br/>[pic_inference.py]"]
  node_layoutreader["LayoutReader"]
end

subgraph group_toolkits["Toolkits and evaluation"]
  node_pfpo_math["Math evaluation data<br/>[math.py]"]
  node_pfpo_apps["Code-task data<br/>[apps.py]"]
  node_pfpo_logger["Training logger<br/>[logger.py]"]
  node_pipelines["Transformers pipelines<br/>[pipelines.py]"]
  node_retrieval_utils["Embedding utilities<br/>[utils.py]"]
  node_aggressive_decoding["Aggressive decoding<br/>[inference.py]"]
end

node_beit_data -->|"imports transforms"| node_beit_transforms
node_beit_data -->|"creates masks"| node_beit_masks
node_multilingual -->|"loads datasets"| node_data_utils
node_multilingual -->|"loads indexed data"| node_indexed_data
node_multilingual -->|"uses sampling"| node_sampled_multi
node_data_utils -->|"builds datasets"| node_indexed_data
node_data_utils -->|"uses path manager"| node_file_io
node_pfpo_apps -->|"uses logger"| node_pfpo_logger
node_latent_inference -->|"uses solver"| node_latent_solver

click node_diff_transformer "https://github.com/microsoft/unilm/blob/master/Diff-Transformer/multihead_diffattn.py"
click node_yoco "https://github.com/microsoft/unilm/blob/master/YOCO/yoco/models/yoco.py"
click node_resa "https://github.com/microsoft/unilm/blob/master/ReSA/llm/arch/model.py"
click node_beit "https://github.com/microsoft/unilm/blob/master/beit/modeling_pretrain.py"
click node_beit_data "https://github.com/microsoft/unilm/blob/master/beit/datasets.py"
click node_beit_transforms "https://github.com/microsoft/unilm/blob/master/beit/transforms.py"
click node_beit_masks "https://github.com/microsoft/unilm/blob/master/beit/masking_generator.py"
click node_beit2 "https://github.com/microsoft/unilm/blob/master/beit2/modeling_vqkd.py"
click node_beit3 "https://github.com/microsoft/unilm/blob/master/beit3/modeling_finetune.py"
click node_multilingual "https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/multilingual/multilingual_data_manager.py"
click node_data_utils "https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/data_utils.py"
click node_indexed_data "https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/indexed_dataset.py"
click node_sampled_multi "https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/multilingual/sampled_multi_dataset.py"
click node_file_io "https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/file_io.py"
click node_adalm "https://github.com/microsoft/unilm/blob/master/adalm/finetune/run_classifier.py"
click node_pfpo_math "https://github.com/microsoft/unilm/blob/master/PFPO/data/math.py"
click node_pfpo_apps "https://github.com/microsoft/unilm/blob/master/PFPO/data/apps.py"
click node_pfpo_logger "https://github.com/microsoft/unilm/blob/master/PFPO/general_util/logger.py"
click node_pipelines "https://github.com/microsoft/unilm/blob/master/xtune/src/transformers/pipelines.py"
click node_retrieval_utils "https://github.com/microsoft/unilm/blob/master/e5/utils.py"
click node_latent_inference "https://github.com/microsoft/unilm/blob/master/LatentLM/inference_speed.py"
click node_latent_solver "https://github.com/microsoft/unilm/blob/master/LatentLM/schedule/dpm_solver.py"
click node_latent_models "https://github.com/microsoft/unilm/blob/master/LatentLM/models/DiT.py"
click node_textdiffuser "https://github.com/microsoft/unilm/blob/master/textdiffuser/inference.py"
click node_textdiffuser2 "https://github.com/microsoft/unilm/blob/master/textdiffuser-2/inference_textdiffuser2_t2i_full.py"
click node_kosmos25 "https://github.com/microsoft/unilm/blob/master/kosmos-2.5/inference.py"
click node_trocr "https://github.com/microsoft/unilm/blob/master/trocr/pic_inference.py"
click node_layoutreader "https://github.com/microsoft/unilm/blob/master/layoutreader/s2s_ft/modeling_decoding.py"
click node_aggressive_decoding "https://github.com/microsoft/unilm/blob/master/decoding/GAD/inference.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_diff_transformer,node_yoco,node_resa toneBlue
class node_beit,node_beit_data,node_beit_transforms,node_beit_masks,node_beit2,node_beit3,node_multilingual,node_data_utils,node_indexed_data,node_sampled_multi,node_file_io,node_adalm toneAmber
class node_latent_inference,node_latent_solver,node_latent_models,node_textdiffuser,node_textdiffuser2,node_kosmos25,node_trocr,node_layoutreader toneMint
class node_pfpo_math,node_pfpo_apps,node_pfpo_logger,node_pipelines,node_retrieval_utils,node_aggressive_decoding toneRose
```

## Components

### Model architectures

- [Diff Transformer](https://github.com/microsoft/unilm/blob/master/Diff-Transformer/multihead_diffattn.py): `Diff-Transformer/multihead_diffattn.py`
- [YOCO](https://github.com/microsoft/unilm/blob/master/YOCO/yoco/models/yoco.py): `YOCO/yoco/models/yoco.py`
- [ReSA LLM](https://github.com/microsoft/unilm/blob/master/ReSA/llm/arch/model.py): `ReSA/llm/arch/model.py`

### Training and data

- [BEiT](https://github.com/microsoft/unilm/blob/master/beit/modeling_pretrain.py): `beit/modeling_pretrain.py`
- [BEiT data](https://github.com/microsoft/unilm/blob/master/beit/datasets.py): `beit/datasets.py`
- [Image transforms](https://github.com/microsoft/unilm/blob/master/beit/transforms.py): `beit/transforms.py`
- [Mask generation](https://github.com/microsoft/unilm/blob/master/beit/masking_generator.py): `beit/masking_generator.py`
- [BEiT-2](https://github.com/microsoft/unilm/blob/master/beit2/modeling_vqkd.py): `beit2/modeling_vqkd.py`
- [BEiT-3](https://github.com/microsoft/unilm/blob/master/beit3/modeling_finetune.py): `beit3/modeling_finetune.py`
- [Multilingual manager](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/multilingual/multilingual_data_manager.py): `edgelm/fairseq/data/multilingual/multilingual_data_manager.py`
- [Dataset utilities](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/data_utils.py): `edgelm/fairseq/data/data_utils.py`
- [Indexed datasets](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/indexed_dataset.py): `edgelm/fairseq/data/indexed_dataset.py`
- [Sampled datasets](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/data/multilingual/sampled_multi_dataset.py): `edgelm/fairseq/data/multilingual/sampled_multi_dataset.py`
- [Path manager](https://github.com/microsoft/unilm/blob/master/edgelm/fairseq/file_io.py): `edgelm/fairseq/file_io.py`
- [AdaLM adaptation](https://github.com/microsoft/unilm/blob/master/adalm/finetune/run_classifier.py): `adalm/finetune/run_classifier.py`

### Applications

- [LatentLM inference](https://github.com/microsoft/unilm/blob/master/LatentLM/inference_speed.py): `LatentLM/inference_speed.py`
- [Diffusion solver](https://github.com/microsoft/unilm/blob/master/LatentLM/schedule/dpm_solver.py): `LatentLM/schedule/dpm_solver.py`
- [LatentLM models](https://github.com/microsoft/unilm/blob/master/LatentLM/models/DiT.py): `LatentLM/models/DiT.py`
- [TextDiffuser](https://github.com/microsoft/unilm/blob/master/textdiffuser/inference.py): `textdiffuser/inference.py`
- [TextDiffuser-2](https://github.com/microsoft/unilm/blob/master/textdiffuser-2/inference_textdiffuser2_t2i_full.py): `textdiffuser-2/inference_textdiffuser2_t2i_full.py`
- [Kosmos-2.5](https://github.com/microsoft/unilm/blob/master/kosmos-2.5/inference.py): `kosmos-2.5/inference.py`
- [TrOCR](https://github.com/microsoft/unilm/blob/master/trocr/pic_inference.py): `trocr/pic_inference.py`
- [LayoutReader](https://github.com/microsoft/unilm/blob/master/layoutreader/s2s_ft/modeling_decoding.py): `layoutreader/s2s_ft/modeling_decoding.py`

### Toolkits and evaluation

- [Math evaluation data](https://github.com/microsoft/unilm/blob/master/PFPO/data/math.py): `PFPO/data/math.py`
- [Code-task data](https://github.com/microsoft/unilm/blob/master/PFPO/data/apps.py): `PFPO/data/apps.py`
- [Training logger](https://github.com/microsoft/unilm/blob/master/PFPO/general_util/logger.py): `PFPO/general_util/logger.py`
- [Transformers pipelines](https://github.com/microsoft/unilm/blob/master/xtune/src/transformers/pipelines.py): `xtune/src/transformers/pipelines.py`
- [Embedding utilities](https://github.com/microsoft/unilm/blob/master/e5/utils.py): `e5/utils.py`
- [Aggressive decoding](https://github.com/microsoft/unilm/blob/master/decoding/GAD/inference.py): `decoding/GAD/inference.py`

## Connections

- BEiT data → Image transforms: imports transforms (evidence: `beit/datasets.py`)
- BEiT data → Mask generation: creates masks (evidence: `beit/datasets.py`)
- Multilingual manager → Dataset utilities: loads datasets (evidence: `edgelm/fairseq/data/multilingual/multilingual_data_manager.py`)
- Multilingual manager → Indexed datasets: loads indexed data (evidence: `edgelm/fairseq/data/multilingual/multilingual_data_manager.py`)
- Multilingual manager → Sampled datasets: uses sampling (evidence: `edgelm/fairseq/data/multilingual/multilingual_data_manager.py`)
- Dataset utilities → Indexed datasets: builds datasets (evidence: `edgelm/fairseq/data/data_utils.py`)
- Dataset utilities → Path manager: uses path manager (evidence: `edgelm/fairseq/data/data_utils.py`)
- Code-task data → Training logger: uses logger (evidence: `PFPO/data/apps.py`)
- LatentLM inference → Diffusion solver: uses solver (evidence: `LatentLM/inference_speed.py`)
