# xiaomi-research/spatio-lm architecture

> Architecture diagram of the GitHub repository xiaomi-research/spatio-lm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaomi-research/spatio-lm
- Repository: https://github.com/xiaomi-research/spatio-lm

## Overview

SpatioLM is a parameter-efficient spatial-intelligence framework for vision-language models. Its main workflow prepares image/video examples, trains with language-model and 3D teacher-distillation objectives, then uses checkpoints for spatial reasoning, perception, or action tasks. The repository also provides a separate benchmark-evaluation stack, including local-model and OpenAI-compatible API evaluation. The README documents these capabilities; sampled source confirms selected model, data, trainer, and evaluation components, so links between unsampled subsystems are included only where documented.

Main entry points: [`src/spatiolm/models/internvl/modeling_internvl3r_chat.py`](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/internvl/modeling_internvl3r_chat.py) (SpatioLM VLM), [`src/spatiolm/swift/trainers.py`](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/swift/trainers.py) (Distillation trainer), [`src/slm_eval/eval_spatial.py`](https://github.com/xiaomi-research/spatio-lm/blob/main/src/slm_eval/eval_spatial.py) (Spatial eval CLI), [`src/spatiolm/datasets/depthlm_data.py`](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/datasets/depthlm_data.py) (DepthLM dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and inputs"]
  node_train_data["Training examples"]
  node_depth_dataset["DepthLM dataset<br/>[depthlm_data.py]"]
  node_video_dataset["Video loader<br/>[video_dataset.py]"]
  node_data_pipelines["Depth transforms"]
  node_templates["Vision templates"]
end

subgraph group_models["Spatial models"]
  node_spatial_model["SpatioLM VLM"]
  node_vision_backbone["Vision encoder"]
  node_qwen_condition["Qwen 3R condition<br/>[qwen3.py]"]
  node_da3_model["DA3 depth model<br/>[modeling_da3.py]"]
  node_vggt_model["VGGT model<br/>[vggt.py]"]
end

subgraph group_training["Training"]
  node_sft_cli["3D SFT entry<br/>[sft.py]"]
  node_trainer["Distillation trainer<br/>[trainers.py]"]
  node_losses["Distillation losses"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_cli["Spatial eval CLI<br/>[eval_spatial.py]"]
  node_benchmark_tasks["Spatial benchmarks"]
  node_api_model["OpenAI API adapter"]
end

node_user(("Engineer"))
node_teacher["DA3 teacher weights"]
node_checkpoints["Model checkpoints"]
node_lmms_eval["LMMs-Eval"]
node_eval_results["Benchmark results"]

node_user -->|"starts training"| node_sft_cli
node_train_data -->|"loads examples"| node_depth_dataset
node_depth_dataset -->|"uses loader"| node_video_dataset
node_depth_dataset -->|"applies transforms"| node_data_pipelines
node_sft_cli -->|"configures trainer"| node_trainer
node_sft_cli -->|"configures inputs"| node_templates
node_trainer -->|"trains model"| node_spatial_model
node_trainer -->|"runs teacher"| node_teacher
node_trainer -->|"computes objectives"| node_losses
node_trainer -->|"saves weights"| node_checkpoints
node_spatial_model -->|"extracts features"| node_vision_backbone
node_spatial_model -.->|"conditions vision"| node_qwen_condition
node_spatial_model -.->|"supports geometry"| node_da3_model
node_spatial_model -.->|"supports geometry"| node_vggt_model
node_user -->|"runs evaluation"| node_eval_cli
node_eval_cli -->|"loads model"| node_checkpoints
node_eval_cli -->|"selects tasks"| node_benchmark_tasks
node_eval_cli -->|"calls evaluator"| node_lmms_eval
node_eval_cli -.->|"creates API model"| node_api_model
node_api_model -->|"returns responses"| node_eval_results
node_lmms_eval -->|"produces scores"| node_eval_results

click node_depth_dataset "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/datasets/depthlm_data.py"
click node_video_dataset "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/datasets/video_dataset.py"
click node_data_pipelines "https://github.com/xiaomi-research/spatio-lm/tree/main/src/spatiolm/datasets/pipelines"
click node_templates "https://github.com/xiaomi-research/spatio-lm/tree/main/src/spatiolm/templates"
click node_sft_cli "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/cli/sft.py"
click node_trainer "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/swift/trainers.py"
click node_losses "https://github.com/xiaomi-research/spatio-lm/tree/main/src/spatiolm/losses"
click node_spatial_model "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/internvl/modeling_internvl3r_chat.py"
click node_vision_backbone "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/internvl/modeling_intern_vit.py"
click node_qwen_condition "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/modules/qwen3.py"
click node_da3_model "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/da3/modeling_da3.py"
click node_vggt_model "https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/vggt/vggt.py"
click node_eval_cli "https://github.com/xiaomi-research/spatio-lm/blob/main/src/slm_eval/eval_spatial.py"
click node_benchmark_tasks "https://github.com/xiaomi-research/spatio-lm/tree/main/src/slm_eval/tasks"
click node_api_model "https://github.com/xiaomi-research/spatio-lm/blob/main/src/slm_eval/simple/parallel_openai_api.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_data,node_depth_dataset,node_video_dataset,node_data_pipelines,node_templates toneBlue
class node_spatial_model,node_vision_backbone,node_qwen_condition,node_da3_model,node_vggt_model toneAmber
class node_sft_cli,node_trainer,node_losses toneMint
class node_eval_cli,node_benchmark_tasks,node_api_model toneRose
class node_user,node_teacher,node_checkpoints,node_lmms_eval,node_eval_results toneIndigo
```

## Components

### Data and inputs

- **Training examples**
- [DepthLM dataset](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/datasets/depthlm_data.py): `src/spatiolm/datasets/depthlm_data.py`
- [Video loader](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/datasets/video_dataset.py): `src/spatiolm/datasets/video_dataset.py`
- [Depth transforms](https://github.com/xiaomi-research/spatio-lm/tree/main/src/spatiolm/datasets/pipelines): `src/spatiolm/datasets/pipelines`
- [Vision templates](https://github.com/xiaomi-research/spatio-lm/tree/main/src/spatiolm/templates): `src/spatiolm/templates`

### Spatial models

- [SpatioLM VLM](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/internvl/modeling_internvl3r_chat.py): `src/spatiolm/models/internvl/modeling_internvl3r_chat.py`
- [Vision encoder](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/internvl/modeling_intern_vit.py): `src/spatiolm/models/internvl/modeling_intern_vit.py`
- [Qwen 3R condition](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/modules/qwen3.py): `src/spatiolm/models/modules/qwen3.py`
- [DA3 depth model](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/da3/modeling_da3.py): `src/spatiolm/models/da3/modeling_da3.py`
- [VGGT model](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/models/vggt/vggt.py): `src/spatiolm/models/vggt/vggt.py`

### Training

- [3D SFT entry](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/cli/sft.py): `src/spatiolm/cli/sft.py`
- [Distillation trainer](https://github.com/xiaomi-research/spatio-lm/blob/main/src/spatiolm/swift/trainers.py): `src/spatiolm/swift/trainers.py`
- [Distillation losses](https://github.com/xiaomi-research/spatio-lm/tree/main/src/spatiolm/losses): `src/spatiolm/losses`

### Evaluation

- [Spatial eval CLI](https://github.com/xiaomi-research/spatio-lm/blob/main/src/slm_eval/eval_spatial.py): `src/slm_eval/eval_spatial.py`
- [Spatial benchmarks](https://github.com/xiaomi-research/spatio-lm/tree/main/src/slm_eval/tasks): `src/slm_eval/tasks`
- [OpenAI API adapter](https://github.com/xiaomi-research/spatio-lm/blob/main/src/slm_eval/simple/parallel_openai_api.py): `src/slm_eval/simple/parallel_openai_api.py`

### Other components

- **Engineer**
- **DA3 teacher weights**
- **Model checkpoints**
- **LMMs-Eval**
- **Benchmark results**

## Connections

- Engineer → 3D SFT entry: starts training
- Training examples → DepthLM dataset: loads examples
- DepthLM dataset → Video loader: uses loader
- DepthLM dataset → Depth transforms: applies transforms
- 3D SFT entry → Distillation trainer: configures trainer
- 3D SFT entry → Vision templates: configures inputs
- Distillation trainer → SpatioLM VLM: trains model
- Distillation trainer → DA3 teacher weights: runs teacher
- Distillation trainer → Distillation losses: computes objectives
- Distillation trainer → Model checkpoints: saves weights
- SpatioLM VLM → Vision encoder: extracts features
- SpatioLM VLM → Qwen 3R condition: conditions vision
- SpatioLM VLM → DA3 depth model: supports geometry
- SpatioLM VLM → VGGT model: supports geometry
- Engineer → Spatial eval CLI: runs evaluation
- Spatial eval CLI → Model checkpoints: loads model
- Spatial eval CLI → Spatial benchmarks: selects tasks
- Spatial eval CLI → LMMs-Eval: calls evaluator
- Spatial eval CLI → OpenAI API adapter: creates API model
- OpenAI API adapter → Benchmark results: returns responses
- LMMs-Eval → Benchmark results: produces scores