How minghanqin/langsplat works
Project type and purpose
• This is a research-oriented 3D reconstruction and rendering system (“LangSplat”) that integrates natural-language features into a Gaussian splatting pipeline.
• It consists of three main pipelines:
1. Preprocessing and language feature extraction
2. Scene-wise language autoencoder
3. LangSplat optimizer and renderer
• It also includes evaluation scripts for 3D localization and segmentation.
File structure
• Top-level scripts and configs:
– preprocess.py, train.py, render.py, convert.py, process.sh, environment.yml
– submodules/ for Git submodules (rasterization, SAM-based segmentation, KNN)
• Core modules:
– autoencoder/ (dataset.py, model.py, train.py, test.py)
– gaussian_renderer/ (network_gui.py)
– lpipsPyTorch/ (perceptual loss modules)
– scene/ (COLMAP loader, cameras, Gaussian model definition)
– eval/ (evaluation scripts and utilities)
– utils/ (camera, graphics, image, loss, system utilities)
• Build/config files: environment.yml, .gitmodules, .gitignore
README for additional insights
• Confirms the three components: optimizer, autoencoder, preprocessing
• Describes data layout and external dependencies: COLMAP outputs, SAM checkpoints, pre-trained models
• Shows pipeline steps (preprocess → autoencoder → LangSplat train → render → eval)
Main components
a. Data Ingestion & Preprocessing
– Raw images directory → preprocess.py
– External tool: “segment-anything-langsplat” (SAM) generates high-dim language features
– Outputs: language_feature/*.npy
b. Scene-wise Language Autoencoder
– Module: autoencoder/train.py & test.py
– Input: language_feature/*.npy
– Output: compressed 3-dim language_feature_dim3/*.npy
c. LangSplat Optimizer (3D Language Gaussian Splatting)
– Script: train.py
– Components: scene/gaussian_model, scene/colmap_loader, lpipsPyTorch loss
– Inputs: COLMAP sparse data, cameras.json, compressed language features
– Output: checkpoint (chkpnt*.pth), point_cloud.ply
d. Renderer
– Script: render.py & gaussian_renderer/network_gui.py
– Input: trained model checkpoint, feature flags
– Output: rendered images or GUI view
e. Evaluation
– Folder: eval/ (eval.sh, evaluate_iou_loc.py, openclip_encoder.py)
– Inputs: rendered outputs, ground-truth labels
Relationships and data flow
• Raw Images → preprocess.py → language_feature (high-dim)
→ autoencoder → language_feature_dim3
→ LangSplat train → Gaussian splat representation + model checkpoint
→ render.py → final images/visualization
→ eval/ → metrics
• Support libraries:
– scene/ for data loaders & camera setups
– lpipsPyTorch for perceptual loss during optimization
– utils/ for helper functions
Architectural patterns and design principles
• Modular pipelines: clear separation of preprocessing, autoencoding, optimization, rendering, evaluation
• Data-centric stages: each stage reads/writes standardized directory structures
• Use of external submodules for extensibility (SAM, rasterization, KNN)
• PyTorch + CUDA for GPU acceleration, with custom C++ extensions