# huang-yh/gaussianformer architecture

> Architecture diagram of the GitHub repository huang-yh/gaussianformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huang-yh/gaussianformer
- Repository: https://github.com/huang-yh/gaussianformer

## Overview

##### Project Type and Purpose
- This is a research-oriented, full-stack deep learning system for 3D semantic occupancy prediction in autonomous driving, implemented in Python and PyTorch. It takes multi-view camera data (from nuScenes) and produces an object-centric Gaussian representation, and optionally dense 3D occupancy grids.

##### File Structure Analysis
- Top-level scripts:
- train.py – orchestrates training.
- eval.py – runs evaluation/inference.
- config/ – all experiment configurations (base, dataset-specific, probabilistic variants).
- dataset/ – data loaders, samplers, transforms, utils for nuScenes & SurroundOcc.
- model/ – the core network, subdivided into:
- backbone/ – image feature extractor (e.g., ResNet/DCNv2).
- encoder/gaussian_encoder/ – transforms 2D features to 3D Gaussians; contains custom CUDA/C++ ops.
- lifter/ – initializes Gaussians from 2D features via distribution modules.
- neck/ – intermediate feature aggregation (optional).
- head/ – refines and outputs Gaussians; local aggregation modules with custom ops.
- segmentor/ – optional BEV or occupancy decoder.
- utils/ – helper functions, safe sparse ops.
- loss/ – multiple loss definitions (occupancy, BCE, Lovász), base classes, utilities.
- misc/ – checkpoint, metrics, TensorBoard wrapper.
- assets/ – README images and logos; docs/ with installation instructions.

##### README Insights
- Describes two variants: GaussianFormer and GaussianFormer-2.
- Points out dependencies on SurroundOcc and SelfOcc frameworks and custom CUDA extensions.

- Data Ingestion & Preparation
– nuScenes raw images + SurroundOcc occupancy labels
– dataset loader, sampler, 3D transforms
- Model Pipeline
1. Image Backbone → 2D feature maps
2. Gaussian Encoder (sparse conv + cross-attention → Gaussian params)
3. Lifter / Init Module (distribution-based initialization)
4. Neck (optional feature fusion)
5. Gaussian Head (refinement & Gaussian superposition)
6. Gaussian-to-Voxel Splatting (CUDA op → dense occupancy grid)
- Loss Computation
– occupancy loss, semantic loss, regularization
- Optimizer & Checkpoint
– parameter update, checkpoint util (misc/checkpoint_util.py)
- Evaluation & Visualization
- Configuration Manager
– Py-Config loader reads config/*.py, sets hyperparameters
- External Services & Dependencies
– PyTorch, CUDA/C++, custom pip-installed ops, nuScenes dataset

1. Data folder → Dataset Module → DataLoader
2. DataLoader → Model Pipeline → Gaussian outputs
3. Gaussian outputs → Loss Module → Scalar losses
4. Loss → Optimizer → Updated Model
5. Checkpoint util saves/loads model
6. eval.py reads checkpoint → Model → Outputs → Metrics

c. Architectural Patterns & Principles
- Modular design: each subdirectory implements a single responsibility (Backbone, Encoder, Head).
- Plugin pattern: custom CUDA ops are packaged with setup.py for on-the-fly build/install.
- Pipeline pattern: clear separation of data preparation, model forward, loss backward, and I/O.
- Config-driven experiments: no hard-coding of hyperparameters – all in config/*.py.

d. Key Technologies and Frameworks
- PyTorch & torch.nn modules
- CUDA/C++ extensions (deformable attention, local aggregation)
- Sparse convolution libraries (e.g., spconv)
- nuScenes SDK for data parsing
- TensorBoard for logging (misc/tb_wrapper.py)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Stores
    RawImages["nuScenes Raw Images"]:::dataStore
    SurroundLabels["SurroundOcc Labels"]:::dataStore
    Checkpoints["Model Checkpoints"]:::dataStore

    %% Configuration Manager
    ConfigManager["Configuration Manager"]:::pythonModule

    %% Data Ingestion & Preparation
    subgraph "Data Ingestion & Preparation"
        direction TB
        DatasetLoader["Dataset Loader"]:::pythonModule
        Sampler["Sampler"]:::pythonModule
        Transforms3D["3D Transforms"]:::pythonModule
        UtilsDS["Dataset Utils"]:::pythonModule
    end

    %% Model Pipeline
    subgraph "Model Pipeline"
        direction TB
        Backbone["Image Backbone"]:::pythonModule
        subgraph "Gaussian Encoder"
            direction TB
            GaussianEncoderPy["Gaussian Encoder (Python)"]:::pythonModule
            CUDAOpsEnc["Encoder CUDA/C++ Ops"]:::cudaOp
        end
        Lifter["Gaussian Lifter"]:::pythonModule
        InitModule["Gaussian Initializer"]:::pythonModule
        Neck["Neck (Feature Fusion)"]:::pythonModule
        subgraph "Gaussian Head"
            direction TB
            HeadPy["Gaussian Head (Python)"]:::pythonModule
            CUDAOpsHead["Head CUDA/C++ Ops"]:::cudaOp
        end
        Segmentor["Optional Segmentor"]:::pythonModule
    end

    %% Loss & Optimization
    subgraph "Training Components"
        direction TB
        LossModule["Loss Computation"]:::pythonModule
        Optimizer["Optimizer"]:::pythonModule
        CheckpointUtil["Checkpoint Utility"]:::pythonModule
    end

    %% Entry Points
    TrainScript["train.py"]:::pythonModule
    EvalScript["eval.py"]:::pythonModule
    VizScript["visualize.py / vis.py"]:::pythonModule

    %% Flows
    RawImages & SurroundLabels -->|load| DatasetLoader
    DatasetLoader -->|"samples"| Sampler
    Sampler -->|"transforms"| Transforms3D
    Transforms3D -->|"utils"| UtilsDS
    UtilsDS -->|"data batch"| ModelPipeline
    ConfigManager -->|"params"| DatasetLoader
    ConfigManager -->|"params"| ModelPipeline

    ModelPipeline -->|"features"| LossModule
    LossModule -->|"loss values"| Optimizer
    Optimizer -->|"updated weights"| CheckpointUtil
    CheckpointUtil -->|save/load| Checkpoints

    TrainScript -->|"runs pipeline"| ModelPipeline
    TrainScript -->|"uses"| Optimizer
    TrainScript -->|"uses"| CheckpointUtil

    EvalScript -->|"loads checkpoint"| ModelPipeline
    EvalScript -->|"outputs metrics"| Checkpoints

    VizScript -->|"loads checkpoint"| ModelPipeline
    VizScript -->|"renders outputs"| Checkpoints

    %% Click Events
    click DatasetLoader "https://github.com/huang-yh/gaussianformer/blob/main/dataset/dataset.py"
    click Sampler "https://github.com/huang-yh/gaussianformer/blob/main/dataset/sampler.py"
    click Transforms3D "https://github.com/huang-yh/gaussianformer/blob/main/dataset/transform_3d.py"
    click UtilsDS "https://github.com/huang-yh/gaussianformer/blob/main/dataset/utils.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/_base_/misc.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/_base_/model.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/_base_/surroundocc.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/nuscenes_gs144000.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/nuscenes_gs25600_solid.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/prob/nuscenes_gs12800.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/prob/nuscenes_gs25600.py"
    click ConfigManager "https://github.com/huang-yh/gaussianformer/blob/main/config/prob/nuscenes_gs6400.py"
    click TrainScript "https://github.com/huang-yh/gaussianformer/blob/main/train.py"
    click EvalScript "https://github.com/huang-yh/gaussianformer/blob/main/eval.py"
    click VizScript "https://github.com/huang-yh/gaussianformer/blob/main/visualize.py"
    click VizScript "https://github.com/huang-yh/gaussianformer/blob/main/vis.py"
    click Backbone "https://github.com/huang-yh/gaussianformer/blob/main/model/backbone/__init__.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/gaussian_encoder.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/anchor_encoder_module.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/deformable_module.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/ffn_module.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/refine_module.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/refine_module_v2.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/spconv3d_module.py"
    click GaussianEncoderPy "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/utils.py"
    click CUDAOpsEnc "https://github.com/huang-yh/gaussianformer/blob/main/model/encoder/gaussian_encoder/ops/setup.py"
    click Lifter "https://github.com/huang-yh/gaussianformer/blob/main/model/lifter/gaussian_lifter.py"
    click Lifter "https://github.com/huang-yh/gaussianformer/blob/main/model/lifter/gaussian_lifter_v2.py"
    click InitModule "https://github.com/huang-yh/gaussianformer/blob/main/model/lifter/gaussian_initializer/resnet_secondfpn.py"
    click Neck "https://github.com/huang-yh/gaussianformer/blob/main/model/neck/__init__.py"
    click HeadPy "https://github.com/huang-yh/gaussianformer/blob/main/model/head/gaussian_head.py"
    click HeadPy "https://github.com/huang-yh/gaussianformer/blob/main/model/head/base_head.py"
    click CUDAOpsHead "https://github.com/huang-yh/gaussianformer/blob/main/model/head/localagg/setup.py"
    click CUDAOpsHead "https://github.com/huang-yh/gaussianformer/blob/main/model/head/localagg_prob/setup.py"
    click CUDAOpsHead "https://github.com/huang-yh/gaussianformer/blob/main/model/head/localagg_prob_fast/setup.py"
    click Segmentor "https://github.com/huang-yh/gaussianformer/blob/main/model/segmentor/bev_segmentor.py"
    click Segmentor "https://github.com/huang-yh/gaussianformer/blob/main/model/segmentor/base_segmentor.py"
    click LossModule "https://github.com/huang-yh/gaussianformer/blob/main/loss/base_loss.py"
    click LossModule "https://github.com/huang-yh/gaussianformer/blob/main/loss/bce_loss.py"
    click LossModule "https://github.com/huang-yh/gaussianformer/blob/main/loss/occupancy_loss.py"
    click LossModule "https://github.com/huang-yh/gaussianformer/blob/main/loss/multi_loss.py"
    click LossModule "https://github.com/huang-yh/gaussianformer/blob/main/loss/utils/lovasz_softmax.py"
    click CheckpointUtil "https://github.com/huang-yh/gaussianformer/blob/main/misc/checkpoint_util.py"
    click Optimizer "https://github.com/huang-yh/gaussianformer/blob/main/misc/metric_util.py"
    click Optimizer "https://github.com/huang-yh/gaussianformer/blob/main/misc/tb_wrapper.py"
    click RawImages "https://github.com/huang-yh/gaussianformer/tree/main/assets/"
    click SurroundLabels "https://github.com/huang-yh/gaussianformer/tree/main/assets/"
    click Checkpoints "https://github.com/huang-yh/gaussianformer/blob/main/misc/checkpoint_util.py"

    %% Styles
    classDef pythonModule fill:#bbdefb,stroke:#0d47a1,color:#0d47a1,stroke-width:1px;
    classDef cudaOp fill:#ffe0b2,stroke:#e65100,color:#e65100,stroke-width:1px,shape:hexagon;
    classDef dataStore fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20,stroke-width:1px;
    classDef hexagon shape:hexagon;
    class datasetLoader,Sampler,Transforms3D,UtilsDS pythonModule
    class RawImages,SurroundLabels,Checkpoints dataStore
    class CUDAOpsEnc,CUDAOpsHead cudaOp
```
