How anjiecheng/spatialrgpt works
Project Type & Purpose
• SpatialRGPT is a research-grade vision–language model system for grounded spatial reasoning.
• It comprises four high-level “subsystems”:
a. Dataset Synthesis Pipeline
b. Core Model Training & Evaluation
c. Pre-trained Model Artifacts & Scripts
d. Gradio Demo Server (inference)
File-Structure Analysis
• Top-level:
• dataset_pipeline/ – code to generate synthetic 2D/3D spatial datasets (configs, processors, visualizers).
• llava/ – the core multimodal model framework: data loaders, model builders, multilingual/vision encoders, training loop and evaluation.
• demo/ – Gradio server and utilities for interactive inference.
• scripts/srgpt/ – training orchestration for various LLM backbones (sheared_3b, llama2_7b, llama3_8b) and evaluation scripts.
• environment_setup.sh – bootstraps conda envs for training vs. demo.
• pyproject.toml, .flake8, pre-commit, etc. – configuration and lint/build metadata.
README Insights
• Describes three training stages (MM-align, pretrain, SFT) facilitated by scripts in scripts/srgpt/<model>/1_mm_align.sh, 2_pretrain.sh, 3_sft.sh.
• Highlights external dependencies needed for demo (SAM-HQ, Depth-Anything).
• Points out evaluation workflows for Region Classification, SpatialRGPT-Bench, general VLM benchmarks.
Components & Interactions
a. Main Components
1. Dataset Pipeline
– Config modules (configs/v2.py, v2_geocalib.py)
– osdsynth processor (region proposals, prompts, depth synthesis)
– Demo images and assets
2. Core Model Framework (llava/)
– llava/data: dataset mixture loader
– llava/model: multimodal encoder, projector, region extractor, LLM adapter
– llava/train: training loop, deepspeed/sequence-parallel, trainer, slurm utils
– llava/eval: region classification, VQA, spatial benchmarks
3. Training Orchestration
– scripts/srgpt/<backbone>/*.sh: stage-wise training scripts
– environment_setup.sh & pyproject.toml for managing dependencies
4. External Services & Dependencies
– Hugging Face (datasets, model hub)
– Depth-Anything repository & weights
– Segment Anything HQ (SAM-HQ) checkpoint
– Detectron2 for region proposals
5. Gradio Demo Server
– demo/gradio_web_server_multi.py
– demo/utils for markdown rendering, SAM integration, SOM visualization
b. Relationships & Data Flow
• Dataset Pipeline outputs synthesized examples → consumed by llava/data loaders during training.
• Training Orchestration scripts invoke llava/train to train multimodal model, storing checkpoints.
• llava/eval loads checkpoints to run benchmarks, writing results to eval_out.
• Demo Server loads trained checkpoints + external model weights (Depth-Anything, SAM-HQ) → serves interactive interface.
c. Architectural Patterns
• Modular pipeline pattern: separate dataset synthesis, model definition, training, evaluation, deployment.
• Plug-in architecture for multimodal encoders and LLM backbones (via builder factories).
• Script-driven training stages orchestrating multi-step workflows.
d. Key Technologies & Libraries
• PyTorch, Deepspeed, Transformers (Hugging Face).
• Gradio for UI.
• Detectron2, Segment Anything (SAM), Depth-Anything for vision preprocessing.
• Hydra-style configs (custom Python modules under configs/).