# minghanqin/langsplat architecture

> Architecture diagram of the GitHub repository minghanqin/langsplat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minghanqin/langsplat
- Repository: https://github.com/minghanqin/langsplat

## Overview

##### Project type and purpose
- This is a research-oriented 3D reconstruction and rendering system (“LangSplat”) that integrates natural-language features into a Gaussian splatting pipeline.
- It consists of three main pipelines:
1. Preprocessing and language feature extraction
2. Scene-wise language autoencoder
3. LangSplat optimizer and renderer
- It also includes evaluation scripts for 3D localization and segmentation.

##### File structure
- Top-level scripts and configs:
– preprocess.py, train.py, render.py, convert.py, process.sh, environment.yml
– submodules/ for Git submodules (rasterization, SAM-based segmentation, KNN)
- Core modules:
– autoencoder/ (dataset.py, model.py, train.py, test.py)
– gaussian_renderer/ (network_gui.py)
– lpipsPyTorch/ (perceptual loss modules)
– scene/ (COLMAP loader, cameras, Gaussian model definition)
– eval/ (evaluation scripts and utilities)
– utils/ (camera, graphics, image, loss, system utilities)
- Build/config files: environment.yml, .gitmodules, .gitignore

##### README for additional insights
- Confirms the three components: optimizer, autoencoder, preprocessing
- Describes data layout and external dependencies: COLMAP outputs, SAM checkpoints, pre-trained models
- Shows pipeline steps (preprocess → autoencoder → LangSplat train → render → eval)

##### Main components
a. Data Ingestion & Preprocessing
– Raw images directory → preprocess.py
– External tool: “segment-anything-langsplat” (SAM) generates high-dim language features
– Outputs: language_feature/*.npy
b. Scene-wise Language Autoencoder
– Module: autoencoder/train.py & test.py
– Input: language_feature/*.npy
– Output: compressed 3-dim language_feature_dim3/*.npy
c. LangSplat Optimizer (3D Language Gaussian Splatting)
– Script: train.py
– Components: scene/gaussian_model, scene/colmap_loader, lpipsPyTorch loss
– Inputs: COLMAP sparse data, cameras.json, compressed language features
– Output: checkpoint (chkpnt*.pth), point_cloud.ply
d. Renderer
– Script: render.py & gaussian_renderer/network_gui.py
– Input: trained model checkpoint, feature flags
– Output: rendered images or GUI view
e. Evaluation
– Folder: eval/ (eval.sh, evaluate_iou_loc.py, openclip_encoder.py)
– Inputs: rendered outputs, ground-truth labels

##### Relationships and data flow
- Raw Images → preprocess.py → language_feature (high-dim)
→ autoencoder → language_feature_dim3
→ LangSplat train → Gaussian splat representation + model checkpoint
→ render.py → final images/visualization
→ eval/ → metrics
- Support libraries:
– scene/ for data loaders & camera setups
– lpipsPyTorch for perceptual loss during optimization
– utils/ for helper functions

##### Architectural patterns and design principles
- Modular pipelines: clear separation of preprocessing, autoencoding, optimization, rendering, evaluation
- Data-centric stages: each stage reads/writes standardized directory structures
- Use of external submodules for extensibility (SAM, rasterization, KNN)
- PyTorch + CUDA for GPU acceleration, with custom C++ extensions

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Preparation Layer
    subgraph "Data Prep" 
        direction TB
        RawImages(["raw_images/"]):::data
        Preprocess["preprocess.py"]:::preproc
        ProcessSh["process.sh"]:::preproc
        SAM[(segment-anything-langsplat)]:::ext
        LanguageFeat(["language_feature/"]):::data
    end

    %% Autoencoder Layer
    subgraph "Autoencoder" 
        direction TB
        AE_Dataset["autoencoder/dataset.py"]:::autoenc
        AE_Model["autoencoder/model.py"]:::autoenc
        AE_Train["autoencoder/train.py"]:::autoenc
        AE_Test["autoencoder/test.py"]:::autoenc
        LanguageFeatDim3(["language_feature_dim3/"]):::data
    end

    %% Optimizer Layer
    subgraph "Optimizer" 
        direction TB
        LS_Train["train.py"]:::optim
        GaussianModel["scene/gaussian_model.py"]:::optim
        COLMAPLoader["scene/colmap_loader.py"]:::optim
        Cameras["scene/cameras.py"]:::optim
        DatasetReaders["scene/dataset_readers.py"]:::optim
        LPIPS_Loss["lpipsPyTorch/modules/lpips.py"]:::optim
        LPIPS_Net["lpipsPyTorch/modules/networks.py"]:::optim
        Checkpoint(["checkpoint.pth"]):::data
        PointCloud(["point_cloud.ply"]):::data
    end

    %% Rendering Layer
    subgraph "Renderer" 
        direction TB
        RenderScript["render.py"]:::rend
        NetworkGUI["gaussian_renderer/network_gui.py"]:::rend
        RenderedImages(["rendered_images/"]):::data
    end

    %% Evaluation Layer
    subgraph "Eval" 
        direction TB
        EvalSh["eval/eval.sh"]:::eval
        EvalIOULoc["eval/evaluate_iou_loc.py"]:::eval
        OpenCLIP["eval/openclip_encoder.py"]:::eval
        EvalUtils["eval/utils.py"]:::eval
        ColorMaps["eval/colormaps.py"]:::eval
        Colors["eval/colors.py"]:::eval
        Metrics(["metrics/"]):::data
    end

    %% Utilities
    subgraph "Utilities" 
        direction TB
        UtilsCam["utils/camera_utils.py"]:::util
        UtilsGen["utils/general_utils.py"]:::util
        UtilsGfx["utils/graphics_utils.py"]:::util
        UtilsImg["utils/image_utils.py"]:::util
        UtilsLoss["utils/loss_utils.py"]:::util
        UtilsSh["utils/sh_utils.py"]:::util
        UtilsSys["utils/system_utils.py"]:::util
    end

    %% Data Flow
    RawImages -->|"images"| Preprocess
    ProcessSh --> Preprocess
    Preprocess -->|".npy"| LanguageFeat
    LanguageFeat --> AE_Dataset
    SAM -.-> Preprocess
    LanguageFeat --> AE_Train
    AE_Train -->|".npy"| LanguageFeatDim3
    AE_Test -->|validation| LanguageFeatDim3
    LanguageFeatDim3 --> LS_Train
    LS_Train --> GaussianModel
    LS_Train --> COLMAPLoader
    LS_Train --> Cameras
    LS_Train --> DatasetReaders
    LS_Train --> LPIPS_Loss
    LS_Train --> LPIPS_Net
    LS_Train -->|".pth"| Checkpoint
    LS_Train -->|".ply"| PointCloud
    Checkpoint --> RenderScript
    PointCloud --> RenderScript
    RenderScript --> NetworkGUI
    NetworkGUI --> RenderedImages
    RenderedImages --> EvalSh
    EvalSh --> EvalIOULoc
    EvalSh --> OpenCLIP
    EvalIOULoc --> Metrics
    OpenCLIP --> Metrics
    ColorMaps --> EvalIOULoc
    Colors --> EvalIOULoc

    %% Click Events
    click Preprocess "https://github.com/minghanqin/langsplat/blob/main/preprocess.py"
    click ProcessSh "https://github.com/minghanqin/langsplat/blob/main/process.sh"
    click SAM "https://github.com/minghanqin/langsplat/tree/main/submodules/segment-anything-langsplat"
    click AE_Dataset "https://github.com/minghanqin/langsplat/blob/main/autoencoder/dataset.py"
    click AE_Model "https://github.com/minghanqin/langsplat/blob/main/autoencoder/model.py"
    click AE_Train "https://github.com/minghanqin/langsplat/blob/main/autoencoder/train.py"
    click AE_Test "https://github.com/minghanqin/langsplat/blob/main/autoencoder/test.py"
    click LS_Train "https://github.com/minghanqin/langsplat/blob/main/train.py"
    click GaussianModel "https://github.com/minghanqin/langsplat/blob/main/scene/gaussian_model.py"
    click COLMAPLoader "https://github.com/minghanqin/langsplat/blob/main/scene/colmap_loader.py"
    click Cameras "https://github.com/minghanqin/langsplat/blob/main/scene/cameras.py"
    click DatasetReaders "https://github.com/minghanqin/langsplat/blob/main/scene/dataset_readers.py"
    click LPIPS_Loss "https://github.com/minghanqin/langsplat/blob/main/lpipsPyTorch/modules/lpips.py"
    click LPIPS_Net "https://github.com/minghanqin/langsplat/blob/main/lpipsPyTorch/modules/networks.py"
    click RenderScript "https://github.com/minghanqin/langsplat/blob/main/render.py"
    click NetworkGUI "https://github.com/minghanqin/langsplat/blob/main/gaussian_renderer/network_gui.py"
    click EvalSh "https://github.com/minghanqin/langsplat/blob/main/eval/eval.sh"
    click EvalIOULoc "https://github.com/minghanqin/langsplat/blob/main/eval/evaluate_iou_loc.py"
    click OpenCLIP "https://github.com/minghanqin/langsplat/blob/main/eval/openclip_encoder.py"
    click EvalUtils "https://github.com/minghanqin/langsplat/blob/main/eval/utils.py"
    click ColorMaps "https://github.com/minghanqin/langsplat/blob/main/eval/colormaps.py"
    click Colors "https://github.com/minghanqin/langsplat/blob/main/eval/colors.py"
    click UtilsCam "https://github.com/minghanqin/langsplat/blob/main/utils/camera_utils.py"
    click UtilsGen "https://github.com/minghanqin/langsplat/blob/main/utils/general_utils.py"
    click UtilsGfx "https://github.com/minghanqin/langsplat/blob/main/utils/graphics_utils.py"
    click UtilsImg "https://github.com/minghanqin/langsplat/blob/main/utils/image_utils.py"
    click UtilsLoss "https://github.com/minghanqin/langsplat/blob/main/utils/loss_utils.py"
    click UtilsSh "https://github.com/minghanqin/langsplat/blob/main/utils/sh_utils.py"
    click UtilsSys "https://github.com/minghanqin/langsplat/blob/main/utils/system_utils.py"

    %% Styles
    classDef data fill:#F2F3F4,stroke:#ABB2B9,stroke-width:1px
    classDef preproc fill:#AED6F1,stroke:#21618C,stroke-width:1px
    classDef autoenc fill:#ABEBC6,stroke:#196F3D,stroke-width:1px
    classDef optim fill:#A9DFBF,stroke:#1D8348,stroke-width:1px
    classDef rend fill:#F5B041,stroke:#B9770E,stroke-width:1px
    classDef eval fill:#D7BDE2,stroke:#6C3483,stroke-width:1px
    classDef util fill:#FDF2E9,stroke:#AF601A,stroke-width:1px
    classDef ext fill:#F9E79F,stroke:#B7950B,stroke-width:1px
```
