# willyzw/hi-slam2 architecture

> Architecture diagram of the GitHub repository willyzw/hi-slam2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/willyzw/hi-slam2
- Repository: https://github.com/willyzw/hi-slam2

## Overview

##### Project type and purpose
- HI-SLAM2 is a research-grade, full-pipeline monocular SLAM system that reconstructs dense 3D geometry using a Gaussian-splatting map representation.
- It integrates deep-learning priors (MiDaS/Omnidata), optical-flow-based tracking (DROID), non-linear optimization, and GPU-accelerated Gaussian splatting rendering.

##### File structure
- Top-level entry points and scripts:
- demo.py, tsdf_integrate.py
- scripts/ (data download, preprocessing, evaluation, run scripts)
- setup.py + environment.yaml (build and dependencies)
- Core Python package: hislam2/
- modules/: neural nets, correlation, feature extraction, GRUs
- midas/: depth/normal prediction models
- gaussian/: Gaussian-map data structures, renderer, OpenGL GUI
- geom/: pose/BA utilities, projection ops
- track_frontend.py, track_backend.py, factor_graph.py, motion_filter.py
- gs_backend.py (Gaussian-map backend)
- Native extensions: src/*.cu, *.cpp (CUDA kernels for correlations, DROID, custom splatting)
- Third-party C++ libs: diff-gaussian-rasterization, eigen, lietorch, simple-knn

##### Extract architecture insights from the README
- Four major phases: Data Preparation → Real-time SLAM Reconstruction → Post-processing (TSDF integration) → Evaluation
- Runtime dependencies: CUDA-11.8 PyTorch, custom CUDA kernels, OpenGL for SLAM GUI

##### Diagramming the system
1. Data Input & Preprocessing
- scripts/preprocess_*.py (COLMAP, frame extraction)
- calib/ & config/
2. Depth & Normal Priors
- hislam2/midas (Omnidata MiDaS models)
3. Frontend Tracking
- hislam2/track_frontend.py
- DROID network (hislam2/modules + src cor­relation kernels)
4. Backend Optimization & Mapping
- hislam2/track_backend.py, factor_graph.py, motion_filter.py
- hislam2/gs_backend.py (Gaussian map builder)
5. Gaussian Splatting Renderer & GUI
- hislam2/gaussian/gui/slam_gui.py
- OpenGL shaders (gau_frag.glsl, gau_vert.glsl)
6. Post-processing
- tsdf_integrate.py (mesh extraction)
7. Evaluation
- scripts/run_replica.py, run_scannet.py, eval_recon

- Raw images → Preprocessing → undistorted, intrinsics →
- Depth/normal network → Priors →
- Feature extractor & correlation kernels → Frame-to-frame flow → Frontend produces keyframes/initial poses →
- Backend consumes keyframes + priors → Factor graph optimize poses & updates Gaussian map →
- Gaussian map → Renderer (SLAM GUI) and optional TSDF integrator → Outputs: camera trajectories, dense meshes, renderings
- Evaluation scripts consume outputs + ground-truth

c. Architectural patterns & design principles
- Modular separation: data prep, neural-net priors, tracking, mapping, rendering
- GPU acceleration: heavy lifting (depth net, corr kernels, splatting) offloaded to CUDA
- Factor-graph optimization in backend for global consistency
- Clear extensibility: config/ and pretrained_models/ for swapping datasets and models

d. Key technologies & libraries
- Python: main control flow, glue code
- PyTorch: MiDaS depth/normal, DROID-Net
- CUDA/C++: correlation kernels (src/*.cu), custom rasterization (thirdparty)
- OpenGL (via glfw/GL) for real-time GUI rendering

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Entry & Setup"
        EP1["demo.py"]:::entry
        EP2["setup.py"]:::entry
        EP3["environment.yaml"]:::entry
    end

    %% Data & Preprocessing
    subgraph "Data & Preprocessing" 
        DP1["calib/"]:::data
        DP2["config/"]:::data
        DP3["scripts/preprocess_owndata.py"]:::data
        DP4["scripts/preprocess_replica.py"]:::data
        DP5["scripts/preprocess_scannet.py"]:::data
    end

    %% Neural-network Priors
    subgraph "Depth & Normal Priors"
        NP1["hislam2/midas/"]:::priors
        NP2["hislam2/midas/base_model.py"]:::priors
        NP3["hislam2/midas/midas_net.py"]:::priors
        NP4["hislam2/midas/omnidata.py"]:::priors
    end

    %% Frontend Tracking
    subgraph "Frontend Tracking"
        FT1["hislam2/track_frontend.py"]:::tracking
        FT2["hislam2/modules/extractor.py"]:::tracking
        FT3["hislam2/modules/corr.py"]:::tracking
        FT4["hislam2/modules/droid_net.py"]:::tracking
    end

    %% CUDA Kernels & C++ Extensions
    subgraph "CUDA Kernels & C++ Extensions"
        CU1["src/correlation_kernels.cu"]:::native
        CU2["src/altcorr_kernel.cu"]:::native
        CU3["src/droid_kernels.cu"]:::native
        CU4["src/droid.cpp"]:::native
        CU5["thirdparty/diff-gaussian-rasterization/"]:::native
    end

    %% Backend Optimization & Mapping
    subgraph "Backend Optimization & Mapping"
        BE1["hislam2/track_backend.py"]:::backend
        BE2["hislam2/factor_graph.py"]:::backend
        BE3["hislam2/motion_filter.py"]:::backend
        BE4["hislam2/gs_backend.py"]:::backend
        BE5["hislam2/pgo_buffer.py"]:::backend
    end

    %% Rendering & GUI
    subgraph "Gaussian Splatting Renderer & GUI"
        RG1["hislam2/gaussian/gui/slam_gui.py"]:::rendering
        RG2["hislam2/gaussian/gui/gl_render/render_ogl.py"]:::rendering
        RG3["hislam2/gaussian/gui/gl_render/shaders/gau_frag.glsl"]:::rendering
        RG4["hislam2/gaussian/gui/gl_render/shaders/gau_vert.glsl"]:::rendering
        RG5["hislam2/gaussian/scene/gaussian_model.py"]:::rendering
    end

    %% Post-processing
    subgraph "Post-processing"
        PP1["tsdf_integrate.py"]:::post
    end

    %% Evaluation
    subgraph "Evaluation"
        EV1["scripts/run_replica.py"]:::eval
        EV2["scripts/run_scannet.py"]:::eval
        EV3["scripts/eval_recon.py"]:::eval
    end

    %% Data Flow
    EP1 --> DP3
    EP1 --> DP4
    EP1 --> DP5
    DP1 --> NP1
    DP2 --> NP1
    DP3 --> FT1
    DP4 --> FT1
    DP5 --> FT1

    NP1 --> NP2
    NP2 --> FT1
    NP3 --> FT1
    NP4 --> FT1

    FT2 --> FT1
    FT3 --> FT1
    FT4 --> FT1
    FT1 --> CU1
    FT1 --> CU2
    FT1 --> CU3

    CU1 --> FT1
    CU2 --> FT1
    CU3 --> FT1
    CU4 --> BE1

    FT1 --> BE1
    NP1 --> BE1

    BE1 --> BE2
    BE2 --> BE3
    BE3 --> BE4
    BE4 --> BE5

    BE4 --> RG1
    BE4 --> PP1

    RG1 --> RG2
    RG2 --> RG3
    RG2 --> RG4
    RG5 --> RG1

    PP1 --> EV1
    PP1 --> EV2
    PP1 --> EV3
    RG1 --> EV1
    RG1 --> EV2

    %% Styles
    classDef entry fill:#fff3cd,stroke:#665500;
    classDef data fill:#cce5ff,stroke:#004085;
    classDef priors fill:#d4edda,stroke:#155724;
    classDef tracking fill:#ffe5b4,stroke:#996600;
    classDef backend fill:#f8d7da,stroke:#721c24;
    classDef rendering fill:#e2d4f7,stroke:#4b0082;
    classDef post fill:#e2e3e5,stroke:#383d41;
    classDef eval fill:#e2e3e5,stroke:#383d41;
    classDef native fill:#ececec,stroke:#6c757d;

    %% Click Events
    click DP1 "https://github.com/willyzw/hi-slam2/tree/main/calib/"
    click DP2 "https://github.com/willyzw/hi-slam2/tree/main/config/"
    click DP3 "https://github.com/willyzw/hi-slam2/blob/main/scripts/preprocess_owndata.py"
    click DP4 "https://github.com/willyzw/hi-slam2/blob/main/scripts/preprocess_replica.py"
    click DP5 "https://github.com/willyzw/hi-slam2/blob/main/scripts/preprocess_scannet.py"
    click NP1 "https://github.com/willyzw/hi-slam2/tree/main/hislam2/midas/"
    click NP2 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/midas/base_model.py"
    click NP3 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/midas/midas_net.py"
    click NP4 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/midas/omnidata.py"
    click FT1 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/track_frontend.py"
    click FT2 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/modules/extractor.py"
    click FT3 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/modules/corr.py"
    click FT4 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/modules/droid_net.py"
    click CU1 "https://github.com/willyzw/hi-slam2/blob/main/src/correlation_kernels.cu"
    click CU2 "https://github.com/willyzw/hi-slam2/blob/main/src/altcorr_kernel.cu"
    click CU3 "https://github.com/willyzw/hi-slam2/blob/main/src/droid_kernels.cu"
    click CU4 "https://github.com/willyzw/hi-slam2/blob/main/src/droid.cpp"
    click CU5 "https://github.com/willyzw/hi-slam2/tree/main/thirdparty/diff-gaussian-rasterization/"
    click BE1 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/track_backend.py"
    click BE2 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/factor_graph.py"
    click BE3 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/motion_filter.py"
    click BE4 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/gs_backend.py"
    click BE5 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/pgo_buffer.py"
    click RG1 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/gaussian/gui/slam_gui.py"
    click RG2 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/gaussian/gui/gl_render/render_ogl.py"
    click RG3 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/gaussian/gui/gl_render/shaders/gau_frag.glsl"
    click RG4 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/gaussian/gui/gl_render/shaders/gau_vert.glsl"
    click RG5 "https://github.com/willyzw/hi-slam2/blob/main/hislam2/gaussian/scene/gaussian_model.py"
    click PP1 "https://github.com/willyzw/hi-slam2/blob/main/tsdf_integrate.py"
    click EV1 "https://github.com/willyzw/hi-slam2/blob/main/scripts/run_replica.py"
    click EV2 "https://github.com/willyzw/hi-slam2/blob/main/scripts/run_scannet.py"
    click EV3 "https://github.com/willyzw/hi-slam2/blob/main/scripts/eval_recon.py"
    click EP1 "https://github.com/willyzw/hi-slam2/blob/main/demo.py"
    click EP2 "https://github.com/willyzw/hi-slam2/blob/main/setup.py"
    click EP3 "https://github.com/willyzw/hi-slam2/blob/main/environment.yaml"
```
