How huang-yh/gaussianformer works
Project Type and Purpose
• This is a research-oriented, full-stack deep learning system for 3D semantic occupancy prediction in autonomous driving, implemented in Python and PyTorch. It takes multi-view camera data (from nuScenes) and produces an object-centric Gaussian representation, and optionally dense 3D occupancy grids.
File Structure Analysis
• Top-level scripts:
• train.py – orchestrates training.
• eval.py – runs evaluation/inference.
• config/ – all experiment configurations (base, dataset-specific, probabilistic variants).
• dataset/ – data loaders, samplers, transforms, utils for nuScenes & SurroundOcc.
• model/ – the core network, subdivided into:
• backbone/ – image feature extractor (e.g., ResNet/DCNv2).
• encoder/gaussian_encoder/ – transforms 2D features to 3D Gaussians; contains custom CUDA/C++ ops.
• lifter/ – initializes Gaussians from 2D features via distribution modules.
• neck/ – intermediate feature aggregation (optional).
• head/ – refines and outputs Gaussians; local aggregation modules with custom ops.
• segmentor/ – optional BEV or occupancy decoder.
• utils/ – helper functions, safe sparse ops.
• loss/ – multiple loss definitions (occupancy, BCE, Lovász), base classes, utilities.
• misc/ – checkpoint, metrics, TensorBoard wrapper.
• assets/ – README images and logos; docs/ with installation instructions.
README Insights
• Describes two variants: GaussianFormer and GaussianFormer-2.
• Points out dependencies on SurroundOcc and SelfOcc frameworks and custom CUDA extensions.
• Data Ingestion & Preparation
– nuScenes raw images + SurroundOcc occupancy labels
– dataset loader, sampler, 3D transforms
• Model Pipeline
1. Image Backbone → 2D feature maps
2. Gaussian Encoder (sparse conv + cross-attention → Gaussian params)
3. Lifter / Init Module (distribution-based initialization)
4. Neck (optional feature fusion)
5. Gaussian Head (refinement & Gaussian superposition)
6. Gaussian-to-Voxel Splatting (CUDA op → dense occupancy grid)
• Loss Computation
– occupancy loss, semantic loss, regularization
• Optimizer & Checkpoint
– parameter update, checkpoint util (misc/checkpoint_util.py)
• Evaluation & Visualization
• Configuration Manager
– Py-Config loader reads config/*.py, sets hyperparameters
• External Services & Dependencies
– PyTorch, CUDA/C++, custom pip-installed ops, nuScenes dataset
1. Data folder → Dataset Module → DataLoader
2. DataLoader → Model Pipeline → Gaussian outputs
3. Gaussian outputs → Loss Module → Scalar losses
4. Loss → Optimizer → Updated Model
5. Checkpoint util saves/loads model
6. eval.py reads checkpoint → Model → Outputs → Metrics
c. Architectural Patterns & Principles
• Modular design: each subdirectory implements a single responsibility (Backbone, Encoder, Head).
• Plugin pattern: custom CUDA ops are packaged with setup.py for on-the-fly build/install.
• Pipeline pattern: clear separation of data preparation, model forward, loss backward, and I/O.
• Config-driven experiments: no hard-coding of hyperparameters – all in config/*.py.
d. Key Technologies and Frameworks
• PyTorch & torch.nn modules
• CUDA/C++ extensions (deformable attention, local aggregation)
• Sparse convolution libraries (e.g., spconv)
• nuScenes SDK for data parsing
• TensorBoard for logging (misc/tb_wrapper.py)