# nslabteam/unilm architecture

> Architecture diagram of the GitHub repository nslabteam/unilm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nslabteam/unilm
- Repository: https://github.com/nslabteam/unilm

## Overview

This repository is a research suite of pretrained-model projects and downstream NLP, vision, speech, and document applications—not one integrated service. The README describes pretraining and adaptation capabilities, while the tree shows separate model implementations, training or inference entrypoints, and shared toolkit code. The graph preserves those documented product branches without implying a shared runtime pipeline. Relationships between projects are included only where the README explicitly describes reuse; several subsystems are represented at their available entrypoint or model-file granularity.

Main entry points: [`layoutreader/run_seq2seq.py`](https://github.com/nslabteam/unilm/blob/master/layoutreader/run_seq2seq.py) (LayoutReader), [`trocr/pic_inference.py`](https://github.com/nslabteam/unilm/blob/master/trocr/pic_inference.py) (TrOCR Inference), [`beit/run_class_finetuning.py`](https://github.com/nslabteam/unilm/blob/master/beit/run_class_finetuning.py) (BEiT Finetuning), [`layoutlm/deprecated/layoutlm/modeling/layoutlm.py`](https://github.com/nslabteam/unilm/blob/master/layoutlm/deprecated/layoutlm/modeling/layoutlm.py) (LayoutLM)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_language["Language Models"]
  node_adalm["AdaLM Adaptation<br/>[run_pico.py]"]
  node_infoxlm["InfoXLM<br/>[infoxlm.py]"]
  node_deltalm["DeltaLM<br/>[deltalm.py]"]
  node_unilm["UniLM Seq2Seq<br/>[run_seq2seq.py]"]
end

subgraph group_vision_speech["Vision And Speech"]
  node_beit_pretrain["BEiT Pretraining"]
  node_beit_finetune["BEiT Finetuning"]
  node_wavlm["WavLM Speech<br/>[WavLM.py]"]
end

subgraph group_documents["Document AI"]
  node_layoutlm["LayoutLM<br/>[layoutlm.py]"]
  node_layoutlmv2["LayoutLMv2"]
  node_layoutxlm["LayoutXLM"]
  node_markuplm["MarkupLM"]
end

subgraph group_toolkit["Training Toolkit"]
  node_xtune["XTune Toolkit<br/>[trainer.py]"]
  node_s2s["Seq2Seq Fine-tuning<br/>[run_seq2seq.py]"]
end

subgraph group_applications["Applications"]
  node_layoutreader["LayoutReader<br/>[run_seq2seq.py]"]
  node_trocr["TrOCR Inference<br/>[pic_inference.py]"]
end

node_researcher(("Researcher"))

node_researcher -->|"adapts models"| node_adalm
node_researcher -->|"pretrains language"| node_infoxlm
node_researcher -->|"pretrains generation"| node_deltalm
node_researcher -->|"uses language models"| node_unilm
node_researcher -->|"pretrains vision"| node_beit_pretrain
node_researcher -->|"finetunes vision"| node_beit_finetune
node_researcher -->|"pretrains speech"| node_wavlm
node_researcher -->|"uses document AI"| node_layoutlm
node_researcher -->|"uses document AI"| node_layoutlmv2
node_researcher -->|"uses multilingual documents"| node_layoutxlm
node_researcher -->|"pretrains markup"| node_markuplm
node_researcher -->|"fine-tunes seq2seq"| node_s2s
node_researcher -->|"detects reading order"| node_layoutreader
node_researcher -->|"recognizes text"| node_trocr
node_layoutreader -->|"built with"| node_layoutlm
node_trocr -->|"uses pretrained vision"| node_beit_finetune

click node_adalm "https://github.com/nslabteam/unilm/blob/master/adalm/finetune/run_pico.py"
click node_infoxlm "https://github.com/nslabteam/unilm/blob/master/infoxlm/src-infoxlm/infoxlm/models/infoxlm.py"
click node_deltalm "https://github.com/nslabteam/unilm/blob/master/deltalm/deltalm/models/deltalm.py"
click node_unilm "https://github.com/nslabteam/unilm/blob/master/unilm-v1/src/biunilm/run_seq2seq.py"
click node_xtune "https://github.com/nslabteam/unilm/blob/master/xtune/src/pequod/training/trainer.py"
click node_beit_pretrain "https://github.com/nslabteam/unilm/blob/master/beit/run_beit_pretraining.py"
click node_beit_finetune "https://github.com/nslabteam/unilm/blob/master/beit/run_class_finetuning.py"
click node_wavlm "https://github.com/nslabteam/unilm/blob/master/wavlm/WavLM.py"
click node_layoutlm "https://github.com/nslabteam/unilm/blob/master/layoutlm/deprecated/layoutlm/modeling/layoutlm.py"
click node_layoutlmv2 "https://github.com/nslabteam/unilm/blob/master/layoutlmft/layoutlmft/models/layoutlmv2/modeling_layoutlmv2.py"
click node_layoutxlm "https://github.com/nslabteam/unilm/blob/master/layoutlmft/layoutlmft/models/layoutxlm/modeling_layoutxlm.py"
click node_markuplm "https://github.com/nslabteam/unilm/blob/master/markuplm/markuplmft/models/markuplm/modeling_markuplm.py"
click node_s2s "https://github.com/nslabteam/unilm/blob/master/s2s-ft/run_seq2seq.py"
click node_layoutreader "https://github.com/nslabteam/unilm/blob/master/layoutreader/run_seq2seq.py"
click node_trocr "https://github.com/nslabteam/unilm/blob/master/trocr/pic_inference.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_adalm,node_infoxlm,node_deltalm,node_unilm toneBlue
class node_beit_pretrain,node_beit_finetune,node_wavlm toneAmber
class node_layoutlm,node_layoutlmv2,node_layoutxlm,node_markuplm toneMint
class node_xtune,node_s2s toneRose
class node_layoutreader,node_trocr,node_researcher toneIndigo
```

## Components

### Language Models

- [AdaLM Adaptation](https://github.com/nslabteam/unilm/blob/master/adalm/finetune/run_pico.py): `adalm/finetune/run_pico.py`
- [InfoXLM](https://github.com/nslabteam/unilm/blob/master/infoxlm/src-infoxlm/infoxlm/models/infoxlm.py): `infoxlm/src-infoxlm/infoxlm/models/infoxlm.py`
- [DeltaLM](https://github.com/nslabteam/unilm/blob/master/deltalm/deltalm/models/deltalm.py): `deltalm/deltalm/models/deltalm.py`
- [UniLM Seq2Seq](https://github.com/nslabteam/unilm/blob/master/unilm-v1/src/biunilm/run_seq2seq.py): `unilm-v1/src/biunilm/run_seq2seq.py`

### Vision And Speech

- [BEiT Pretraining](https://github.com/nslabteam/unilm/blob/master/beit/run_beit_pretraining.py): `beit/run_beit_pretraining.py`
- [BEiT Finetuning](https://github.com/nslabteam/unilm/blob/master/beit/run_class_finetuning.py): `beit/run_class_finetuning.py`
- [WavLM Speech](https://github.com/nslabteam/unilm/blob/master/wavlm/WavLM.py): `wavlm/WavLM.py`

### Document AI

- [LayoutLM](https://github.com/nslabteam/unilm/blob/master/layoutlm/deprecated/layoutlm/modeling/layoutlm.py): `layoutlm/deprecated/layoutlm/modeling/layoutlm.py`
- [LayoutLMv2](https://github.com/nslabteam/unilm/blob/master/layoutlmft/layoutlmft/models/layoutlmv2/modeling_layoutlmv2.py): `layoutlmft/layoutlmft/models/layoutlmv2/modeling_layoutlmv2.py`
- [LayoutXLM](https://github.com/nslabteam/unilm/blob/master/layoutlmft/layoutlmft/models/layoutxlm/modeling_layoutxlm.py): `layoutlmft/layoutlmft/models/layoutxlm/modeling_layoutxlm.py`
- [MarkupLM](https://github.com/nslabteam/unilm/blob/master/markuplm/markuplmft/models/markuplm/modeling_markuplm.py): `markuplm/markuplmft/models/markuplm/modeling_markuplm.py`

### Training Toolkit

- [XTune Toolkit](https://github.com/nslabteam/unilm/blob/master/xtune/src/pequod/training/trainer.py): `xtune/src/pequod/training/trainer.py`
- [Seq2Seq Fine-tuning](https://github.com/nslabteam/unilm/blob/master/s2s-ft/run_seq2seq.py): `s2s-ft/run_seq2seq.py`

### Applications

- [LayoutReader](https://github.com/nslabteam/unilm/blob/master/layoutreader/run_seq2seq.py): `layoutreader/run_seq2seq.py`
- [TrOCR Inference](https://github.com/nslabteam/unilm/blob/master/trocr/pic_inference.py): `trocr/pic_inference.py`

### Other components

- **Researcher**

## Connections

- Researcher → AdaLM Adaptation: adapts models
- Researcher → InfoXLM: pretrains language
- Researcher → DeltaLM: pretrains generation
- Researcher → UniLM Seq2Seq: uses language models
- Researcher → BEiT Pretraining: pretrains vision
- Researcher → BEiT Finetuning: finetunes vision
- Researcher → WavLM Speech: pretrains speech
- Researcher → LayoutLM: uses document AI
- Researcher → LayoutLMv2: uses document AI
- Researcher → LayoutXLM: uses multilingual documents
- Researcher → MarkupLM: pretrains markup
- Researcher → Seq2Seq Fine-tuning: fine-tunes seq2seq
- Researcher → LayoutReader: detects reading order
- Researcher → TrOCR Inference: recognizes text
- LayoutReader → LayoutLM: built with
- TrOCR Inference → BEiT Finetuning: uses pretrained vision