How xiaomi-research/spatio-lm works
SpatioLM is a parameter-efficient spatial-intelligence framework for vision-language models. Its main workflow prepares image/video examples, trains with language-model and 3D teacher-distillation objectives, then uses checkpoints for spatial reasoning, perception, or action tasks. The repository also provides a separate benchmark-evaluation stack, including local-model and OpenAI-compatible API evaluation. The README documents these capabilities; sampled source confirms selected model, data, trainer, and evaluation components, so links between unsampled subsystems are included only where documented.
Main entry points: modeling_internvl3r_chat.py SpatioLM VLM · trainers.py Distillation trainer · eval_spatial.py Spatial eval CLI · depthlm_data.py DepthLM dataset
Components 21
Data and inputs
- Training examples
- DepthLM dataset
src/spatiolm/datasets/depthlm_data.py - Video loader
src/spatiolm/datasets/video_dataset.py - Depth transforms
src/spatiolm/datasets/pipelines - Vision templates
src/spatiolm/templates
Spatial models
- SpatioLM VLM
src/spatiolm/models/internvl/modeling_internvl3r_chat.py - Vision encoder
src/spatiolm/models/internvl/modeling_intern_vit.py - Qwen 3R condition
src/spatiolm/models/modules/qwen3.py - DA3 depth model
src/spatiolm/models/da3/modeling_da3.py - VGGT model
src/spatiolm/models/vggt/vggt.py
Training
- 3D SFT entry
src/spatiolm/cli/sft.py - Distillation trainer
src/spatiolm/swift/trainers.py - Distillation losses
src/spatiolm/losses
Evaluation
- Spatial eval CLI
src/slm_eval/eval_spatial.py - Spatial benchmarks
src/slm_eval/tasks - OpenAI API adapter
src/slm_eval/simple/parallel_openai_api.py
Other components
- Engineer
- DA3 teacher weights
- Model checkpoints
- LMMs-Eval
- Benchmark results
Connections 21
- Engineer → 3D SFT entry: starts training
- Training examples → DepthLM dataset: loads examples
- DepthLM dataset → Video loader: uses loader
- DepthLM dataset → Depth transforms: applies transforms
- 3D SFT entry → Distillation trainer: configures trainer
- 3D SFT entry → Vision templates: configures inputs
- Distillation trainer → SpatioLM VLM: trains model
- Distillation trainer → DA3 teacher weights: runs teacher
- Distillation trainer → Distillation losses: computes objectives
- Distillation trainer → Model checkpoints: saves weights
- SpatioLM VLM → Vision encoder: extracts features
- SpatioLM VLM → Qwen 3R condition: conditions vision
- SpatioLM VLM → DA3 depth model: supports geometry
- SpatioLM VLM → VGGT model: supports geometry
- Engineer → Spatial eval CLI: runs evaluation
- Spatial eval CLI → Model checkpoints: loads model
- Spatial eval CLI → Spatial benchmarks: selects tasks
- Spatial eval CLI → LMMs-Eval: calls evaluator
- Spatial eval CLI → OpenAI API adapter: creates API model
- OpenAI API adapter → Benchmark results: returns responses
- LMMs-Eval → Benchmark results: produces scores