# anjiecheng/spatialrgpt architecture

> Architecture diagram of the GitHub repository anjiecheng/spatialrgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anjiecheng/spatialrgpt
- Repository: https://github.com/anjiecheng/spatialrgpt

## Overview

##### Project Type & Purpose
- SpatialRGPT is a research-grade vision–language model system for grounded spatial reasoning.
- It comprises four high-level “subsystems”:
a. Dataset Synthesis Pipeline
b. Core Model Training & Evaluation
c. Pre-trained Model Artifacts & Scripts
d. Gradio Demo Server (inference)

##### File-Structure Analysis
- Top-level:
- dataset_pipeline/ – code to generate synthetic 2D/3D spatial datasets (configs, processors, visualizers).
- llava/ – the core multimodal model framework: data loaders, model builders, multilingual/vision encoders, training loop and evaluation.
- demo/ – Gradio server and utilities for interactive inference.
- scripts/srgpt/ – training orchestration for various LLM backbones (sheared_3b, llama2_7b, llama3_8b) and evaluation scripts.
- environment_setup.sh – bootstraps conda envs for training vs. demo.
- pyproject.toml, .flake8, pre-commit, etc. – configuration and lint/build metadata.

##### README Insights
- Describes three training stages (MM-align, pretrain, SFT) facilitated by scripts in scripts/srgpt/<model>/1_mm_align.sh, 2_pretrain.sh, 3_sft.sh.
- Highlights external dependencies needed for demo (SAM-HQ, Depth-Anything).
- Points out evaluation workflows for Region Classification, SpatialRGPT-Bench, general VLM benchmarks.

##### Components & Interactions
a. Main Components
1. Dataset Pipeline
– Config modules (configs/v2.py, v2_geocalib.py)
– osdsynth processor (region proposals, prompts, depth synthesis)
– Demo images and assets
2. Core Model Framework (llava/)
– llava/data: dataset mixture loader
– llava/model: multimodal encoder, projector, region extractor, LLM adapter
– llava/train: training loop, deepspeed/sequence-parallel, trainer, slurm utils
– llava/eval: region classification, VQA, spatial benchmarks
3. Training Orchestration
– scripts/srgpt/<backbone>/*.sh: stage-wise training scripts
– environment_setup.sh & pyproject.toml for managing dependencies
4. External Services & Dependencies
– Hugging Face (datasets, model hub)
– Depth-Anything repository & weights
– Segment Anything HQ (SAM-HQ) checkpoint
– Detectron2 for region proposals
5. Gradio Demo Server
– demo/gradio_web_server_multi.py
– demo/utils for markdown rendering, SAM integration, SOM visualization

b. Relationships & Data Flow
- Dataset Pipeline outputs synthesized examples → consumed by llava/data loaders during training.
- Training Orchestration scripts invoke llava/train to train multimodal model, storing checkpoints.
- llava/eval loads checkpoints to run benchmarks, writing results to eval_out.
- Demo Server loads trained checkpoints + external model weights (Depth-Anything, SAM-HQ) → serves interactive interface.

c. Architectural Patterns
- Modular pipeline pattern: separate dataset synthesis, model definition, training, evaluation, deployment.
- Plug-in architecture for multimodal encoders and LLM backbones (via builder factories).
- Script-driven training stages orchestrating multi-step workflows.

d. Key Technologies & Libraries
- PyTorch, Deepspeed, Transformers (Hugging Face).
- Gradio for UI.
- Detectron2, Segment Anything (SAM), Depth-Anything for vision preprocessing.
- Hydra-style configs (custom Python modules under configs/).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DP["Dataset Synthesis Pipeline"]:::internal
        DP_configs["configs (v2.py, v2_geocalib.py)"]:::internal
        DP_processor["osdsynth processor"]:::internal
        DP_visualizer["osdsynth visualizer (som.py)"]:::internal
        DP_scripts["weight scripts"]:::script
        SynthData[(Synthetic Dataset)]:::data
        DP --> DP_configs
        DP --> DP_processor
        DP --> DP_visualizer
        DP --> DP_scripts
        DP -->|“generates”| SynthData
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        subgraph "Core Model Framework"
            direction TB
            DataLoader["llava/data"]:::internal
            subgraph "Model Components"
                direction TB
                Encoder["multimodal_encoder"]:::internal
                Projector["multimodal_projector"]:::internal
                RegionExt["region_extractor"]:::internal
                Builder["builder.py"]:::internal
                ConfigLM["configuration_llava.py"]:::internal
                LMAdapter["language_model/"]:::internal
            end
            subgraph "Training Engine"
                direction TB
                TrainLoop["train.py"]:::internal
                Trainer["llava_trainer.py"]:::internal
                Slurm["slurm_utils.py"]:::internal
                DeepSpeed["deepspeed_replace/"]:::internal
                SeqPar["sequence_parallel/"]:::internal
            end
        end
        Orchestrator["Training Orchestration Scripts"]:::script
        EnvSetup["environment_setup.sh & pyproject.toml"]:::script
        Checkpoints[(Model Checkpoints)]:::data
        SynthData --> DataLoader
        DataLoader --> TrainLoop
        TrainLoop -->|“trains model”| Checkpoints
        Orchestrator --> TrainLoop
        EnvSetup --> Orchestrator
        TrainLoop --> Trainer
        TrainLoop --> Slurm
        TrainLoop --> DeepSpeed
        TrainLoop --> SeqPar
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        subgraph "Eval Framework"
            direction TB
            EvalRegion["eval_region_cls.py"]:::internal
            EvalSpatial["eval_spatial.py"]:::internal
            EvalVQA["model_vqa.py"]:::internal
            EvalQA["model_qa.py"]:::internal
        end
        EvalScripts["Eval Orchestration & Templates"]:::script
        Results[(Benchmark Results)]:::data
        Checkpoints --> EvalRegion
        Checkpoints --> EvalSpatial
        Checkpoints --> EvalVQA
        Checkpoints --> EvalQA
        EvalRegion --> Results
        EvalSpatial --> Results
        EvalVQA --> Results
        EvalQA --> Results
        Results --> EvalScripts
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        DemoServer["gradio_web_server_multi.py"]:::internal
        subgraph "Demo Utilities"
            direction TB
            MDUtils["markdown.py"]:::internal
            SAMUtils["sam_utils.py"]:::internal
            SOMUtils["som.py"]:::internal
        end
        DemoServer --> MDUtils
        DemoServer --> SAMUtils
        DemoServer --> SOMUtils
        Checkpoints --> DemoServer
        SAMExt["Segment Anything HQ"]:::external
        DepthExt["Depth-Anything"]:::external
        HFExt["Hugging Face"]:::external
        DetectExt["Detectron2"]:::external
        SAMExt -->|“checkpoint”| DemoServer
        DepthExt -->|“weights”| DemoServer
        HFExt -->|“models/datasets”| DemoServer
        DetectExt -->|“region proposals”| DemoServer
    end

    %% Legend / Environments Note
    subgraph "Environments"
        direction TB
        note1["Training Env (conda)"]:::note
        note2["Demo Env (conda)"]:::note
    end

    classDef internal fill:#cce5ff,stroke:#004085,color:#004085;
    classDef data fill:#d4edda,stroke:#155724,color:#155724,shape:cylinder;
    classDef external fill:#fff3cd,stroke:#856404,color:#856404;
    classDef script fill:#ffe5b4,stroke:#cc8400,color:#cc8400,stroke-dasharray: 5 5,shape:rectRounded;
    classDef note fill:#f8f9fa,stroke:#6c757d,color:#6c757d,shape:note;

    %% Click Events
    click DP "https://github.com/anjiecheng/spatialrgpt/tree/main/dataset_pipeline/"
    click DP_configs "https://github.com/anjiecheng/spatialrgpt/blob/main/dataset_pipeline/configs/v2.py"
    click DP_configs "https://github.com/anjiecheng/spatialrgpt/blob/main/dataset_pipeline/configs/v2_geocalib.py"
    click DP_processor "https://github.com/anjiecheng/spatialrgpt/tree/main/dataset_pipeline/osdsynth/processor/"
    click DP_visualizer "https://github.com/anjiecheng/spatialrgpt/blob/main/dataset_pipeline/osdsynth/visualizer/som.py"
    click DP_scripts "https://github.com/anjiecheng/spatialrgpt/blob/main/dataset_pipeline/scripts/download_all_weights.sh"
    click DP_scripts "https://github.com/anjiecheng/spatialrgpt/blob/main/dataset_pipeline/scripts/remove_all_weights.sh"
    click DataLoader "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/data/"
    click Encoder "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/model/multimodal_encoder/"
    click Projector "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/model/multimodal_projector/"
    click RegionExt "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/model/region_extractor/"
    click Builder "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/model/builder.py"
    click ConfigLM "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/model/configuration_llava.py"
    click LMAdapter "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/model/language_model/"
    click TrainLoop "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/train/train.py"
    click Trainer "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/train/llava_trainer.py"
    click Slurm "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/train/slurm_utils.py"
    click DeepSpeed "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/train/deepspeed_replace/"
    click SeqPar "https://github.com/anjiecheng/spatialrgpt/tree/main/llava/train/sequence_parallel/"
    click EvalRegion "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/eval/eval_region_cls.py"
    click EvalSpatial "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/eval/eval_spatial.py"
    click EvalVQA "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/eval/model_vqa.py"
    click EvalQA "https://github.com/anjiecheng/spatialrgpt/blob/main/llava/eval/model_qa.py"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/sheared_3b/1_mm_align.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/sheared_3b/2_pretrain.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/sheared_3b/3_sft.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/llama2_7b/1_mm_align.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/llama2_7b/2_pretrain.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/llama2_7b/3_sft.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/llama3_8b/1_mm_align.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/llama3_8b/2_pretrain.sh"
    click Orchestrator "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/llama3_8b/3_sft.sh"
    click EvalScripts "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/eval/coco_cls.sh"
    click EvalScripts "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/eval/eval_coco_obo.py"
    click EvalScripts "https://github.com/anjiecheng/spatialrgpt/blob/main/scripts/srgpt/eval/srgpt_bench.sh"
    click EvalScripts "https://github.com/anjiecheng/spatialrgpt/tree/main/scripts/srgpt/eval/prompt_templates/"
    click EnvSetup "https://github.com/anjiecheng/spatialrgpt/blob/main/environment_setup.sh"
    click EnvSetup "https://github.com/anjiecheng/spatialrgpt/blob/main/pyproject.toml"
    click DemoServer "https://github.com/anjiecheng/spatialrgpt/blob/main/demo/gradio_web_server_multi.py"
    click MDUtils "https://github.com/anjiecheng/spatialrgpt/blob/main/demo/utils/markdown.py"
    click SAMUtils "https://github.com/anjiecheng/spatialrgpt/blob/main/demo/utils/sam_utils.py"
    click SOMUtils "https://github.com/anjiecheng/spatialrgpt/blob/main/demo/utils/som.py"
```
