# manycore-research/spatiallm architecture

> Architecture diagram of the GitHub repository manycore-research/spatiallm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/manycore-research/spatiallm
- Repository: https://github.com/manycore-research/spatiallm

## Overview

##### Project Type and Purpose
- This is a research-oriented, machine learning project built in Python that implements a 3D large language model.
- It processes point clouds (input) and generates structured layout representations (output) for applications in robotics, navigation, and scene understanding.
- The README emphasizes inference, visualization, and evaluation steps as well as providing example usage and benchmarks.

##### File Structure
- Top-level scripts:
– inference.py: orchestrates the inference pipeline (reading point clouds, invoking the model, and writing layout outputs).
– eval.py: evaluates model predictions against ground truth.
- Project Modules (inside the "spatiallm" package):
– layout (entity.py, layout.py): Handles processing and representation of architectural layout information.
– model (pcd_encoder.py, spatiallm_llama.py, spatiallm_qwen.py): Implements different model variants and encoders.
– pcd (pcd_loader.py, registry.py): Contains functionality for loading and managing point cloud data.
- Configuration and build files: .gitignore, pyproject.toml, and supplementary docs (EXAMPLE.md, LICENSE.txt, code_template.txt).

##### Key Components of the System
A. Data Input and Preprocessing:
- Point Cloud Input: Loaded using modules in spatiallm/pcd (pcd_loader.py, registry.py).
- External data sources: Mention of MASt3R-SLAM outputs and HuggingFace datasets provides context for data ingestion.
B. Model Processing:
- PCD Encoder: In spatiallm/model/pcd_encoder.py, which processes raw point clouds.
- SpatialLM Model Variants: The two provided implementations (spatiallm_llama.py and spatiallm_qwen.py) indicate different model architectures based on Llama and Qwen variants.
C. Layout and Semantic Parsing:
- Layout module: Handles the conversion from encoded representations to structured scene layouts (spatiallm/layout/layout.py and entity.py).
D. Inference and Evaluation Pipelines:
- Inference Script (inference.py): Connects data input, model invocation, and layout generation.
- Evaluation Script (eval.py): Compares predicted layouts (e.g., via F1 scores) with ground truth.
E. Visualization:

##### Relationships and Interactions
- Input point clouds are first loaded and preprocessed, then passed to the PCD encoder.
- The encoded representation flows into the model for spatial understanding.
- Model outputs are then processed by the layout module to format a structured scene layout.
- Separate pipelines orchestrated through distinct scripts invoke the above modules:
– Inference script controls the overall flow.
– Evaluation script uses outputs and compares them to benchmark datasets.
– Visualization script formats outputs for visual analysis.
- External dependencies and services (HuggingFace, rerun, and SLAM systems like MASt3R-SLAM or SLAM3R) interact with this core pipeline.

##### What
- Clear labels for:
– Input: “Point Cloud Data (e.g., PLY files)”
– Modules: “PCD Loader & Preprocessing”, “PCD Encoder”, “SpatialLM Model (LLama / Qwen)”, “Layout Generation”
– External dependencies: “Hugging Face Datasets/Models”, “Rerun (Visualization)”, “SLAM Inputs (MASt3R-SLAM/SLAM3R)”
- Annotations that highlight which parts of the system are configurable (e.g., command-line arguments in scripts) or which components could potentially be extended in the future.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Scripts
    subgraph "Orchestration"
        pipeline_inference("inference.py"):::script
        pipeline_eval("eval.py"):::script
        pipeline_visualize("visualize.py"):::script
    end

    %% Data Input & Preprocessing Layer
    subgraph "Data Input & Preprocessing"
        data_in("Point Cloud Data (PLY)"):::data
        pcd_loader("PCD Loader (pcd_loader.py)"):::data
        registry("Registry (registry.py)"):::data
    end

    %% Model Processing Layer
    subgraph "Model Processing"
        encoder("PCD Encoder (pcd_encoder.py)"):::model
        llama("SpatialLM-Llama-1B (spatiallm_llama.py)"):::model
        qwen("SpatialLM-Qwen-0.5B (spatiallm_qwen.py)"):::model
    end

    %% Layout Processing Layer
    subgraph "Layout Processing"
        layout_mod("Layout Module (layout.py)"):::layout
        entity_mod("Entity Module (entity.py)"):::layout
        layout_proc("Layout Generation"):::layout
    end

    %% External Dependencies (for reference)
    external_dep("External Services\\nHuggingFace,Datasets;Rerun;SLAM Inputs"):::external

    %% Data Flow Connections
    pipeline_inference -->|"starts"| data_in
    data_in -->|"loads via"| pcd_loader
    pcd_loader -->|"registers to"| registry
    registry -->|"passes data to"| encoder
    encoder -->|"selects variant"| llama
    encoder -->|"selects variant"| qwen
    llama -->|"outputs to"| layout_mod
    qwen -->|"outputs to"| layout_mod
    layout_mod -->|"combined with"| entity_mod
    entity_mod -->|"generates"| layout_proc

    %% Pipeline Outputs: Evaluated and Visualized
    layout_proc -->|"used by"| pipeline_eval
    layout_proc -->|"used by"| pipeline_visualize
    layout_proc -->|"also returned to"| pipeline_inference

    %% External dependencies connected to orchestration and visualization
    pipeline_inference --- external_dep
    pipeline_visualize --- external_dep

    %% Click Events for Component Mapping
    click pcd_loader "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/pcd/pcd_loader.py"
    click registry "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/pcd/registry.py"
    click encoder "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/model/pcd_encoder.py"
    click llama "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/model/spatiallm_llama.py"
    click qwen "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/model/spatiallm_qwen.py"
    click layout_mod "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/layout/layout.py"
    click entity_mod "https://github.com/manycore-research/spatiallm/blob/main/spatiallm/layout/entity.py"
    click pipeline_inference "https://github.com/manycore-research/spatiallm/blob/main/inference.py"
    click pipeline_eval "https://github.com/manycore-research/spatiallm/blob/main/eval.py"
    click pipeline_visualize "https://github.com/manycore-research/spatiallm/blob/main/visualize.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef model fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef layout fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef script fill:#EE82EE,stroke:#333,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px;
```
