How hustvl/4dgaussians works
Purpose and execution surface
This is a PyTorch research pipeline for reconstructing and real-time rendering dynamic scenes as time-conditioned 4D Gaussian Splatting models. Primary CLI entry points are train.py (optimization), render.py (checkpoint rendering), metrics.py (evaluation), and full_eval.py (batch evaluation). Auxiliary model transformations are export_perframe_3DGS.py and merge_many_4dgs.py.
Dataset ingestion and scene normalization
Dataset dispatch converts D-NeRF, HyperNeRF, DyNeRF/Neural 3D Video, COLMAP, and multi-camera layouts into a common camera/time/point-cloud scene representation. Readers handle source-specific poses, images, frame time, and initialization geometry; camera utilities adapt these for training and rendering.
• scene/dataset_readers.py
• scene/dataset.py
• scene/hyper_loader.py
• scene/neural_3D_dataset_NDC.py
• scene/multipleview_dataset.py
Gaussian scene model
The learned state is a set of canonical 3D Gaussians with appearance and geometric attributes, initialized from SfM/dense point clouds where available. It owns optimization-facing parameters, persistence/checkpoint behavior, and Gaussian lifecycle operations such as densification/pruning implied by the training pipeline.
• scene/gaussian_model.py
• scene/__init__.py
• utils/point_utils.py
Time-dependent deformation field
Dynamic motion is represented separately from canonical Gaussians: a neural deformation module maps spatial position and timestamp to changes used at render time. Its feature representation is based on multi-resolution HexPlane/grid components, with regularization controlling the learned spatiotemporal field.
• scene/deformation.py
• scene/hexplane.py
• scene/grid.py
• scene/regulation.py
Differentiable rendering boundary
The renderer combines a camera, timestamp, canonical Gaussian model, and deformation field, then delegates projection/rasterization to the CUDA rasterizer. Rendered images and auxiliary visibility/radii outputs feed losses during training or are persisted during offline rendering. A socket-oriented GUI layer can expose rendering interactively.
• gaussian_renderer/__init__.py
• gaussian_renderer/network_gui.py
• submodules/depth-diff-gaussian-rasterization
• submodules/simple-knn
Training and experiment orchestration
train.py parses shared/model/optimization arguments, loads a arguments/... scene preset, constructs scene and model state, and runs coarse/fine optimization stages. It writes experiment artifacts under the requested model/output path and supports checkpoint restart. Image reconstruction and perceptual losses come from reusable utility modules and the embedded LPIPS implementation.
• train.py
• arguments/__init__.py
• utils/loss_utils.py
• lpipsPyTorch/modules/lpips.py
Data preparation, outputs, and evaluation
External preprocessing establishes camera calibration and point-cloud initialization: COLMAP is invoked through colmap.sh; format-specific scripts extract frames or convert layouts. Training checkpoints are consumed by rendering and metrics workflows; rendered predictions are compared against dataset imagery using quality metrics. The project targets Python 3.7, PyTorch/CUDA (documented environment: PyTorch 1.13.1 + CUDA 11.6), COLMAP, and optional Nerfstudio preprocessing.
• colmap.sh
• scripts/preprocess_dynerf.py
• scripts/downsample_point.py
• render.py
• metrics.py
Main entry points: train.py Training CLI · __init__.py Differentiable renderer · dataset_readers.py Dataset readers · deformation.py Deformation field
Components 19
Preparation
- Dynamic scene data · D-NeRF, HyperNeRF, DyNeRF, COLMAP, and multi-camera inputs.
- Preprocessing · Extracts frames, calibrates cameras, and prepares point-cloud initialization.
colmap.sh
Scene & Models
- Dataset readers · Normalizes source layouts into cameras, timestamps, and scene geometry.
scene/dataset_readers.py - Scene dataset
scene/dataset.py - Camera adapters · Adapts source cameras for optimization and rendering.
scene/hyper_loader.py - Canonical Gaussians · Owns Gaussian parameters, checkpoints, densification, and pruning.
scene/gaussian_model.py - Deformation field · Maps canonical positions and timestamps to dynamic Gaussian changes.
scene/deformation.py - HexPlane grids · Multi-resolution feature representation for deformation.
scene/hexplane.py - Field regularization
scene/regulation.py
Training & Rendering
- Training CLI · Runs coarse and fine optimization and supports restart checkpoints.
train.py - Arguments & presets
arguments/__init__.py - Differentiable renderer · Combines camera, time, Gaussians, and deformation into render outputs.
gaussian_renderer/__init__.py - CUDA rasterizer
submodules/depth-diff-gaussian-rasterization - Reconstruction losses · Image and perceptual losses used for optimization.
utils/loss_utils.py
Outputs & Evaluation
- Model artifacts · Experiment outputs at the requested model path.
- Render CLI
render.py - Rendered predictions
- Metrics CLI · Compares predictions with dataset imagery.
metrics.py - Batch evaluation
full_eval.py
Connections 25
- Dynamic scene data → Preprocessing: prepare
- Preprocessing → Dataset readers: calibrated inputs
- Dynamic scene data → Dataset readers: source layouts
- Dataset readers → Scene dataset: common scene data
- Dataset readers → Camera adapters: poses and frames
- Scene dataset → Canonical Gaussians: initial point cloud
- HexPlane grids → Deformation field: features
- Field regularization → Deformation field: constrains
- Arguments & presets → Training CLI: optimization settings
- Scene dataset → Training CLI: cameras and times
- Training CLI → Canonical Gaussians: optimizes
- Training CLI → Deformation field: optimizes
- Camera adapters → Differentiable renderer: camera and timestamp
- Canonical Gaussians → Differentiable renderer: canonical state
- Deformation field → Differentiable renderer: dynamic offsets
- Differentiable renderer → CUDA rasterizer: project and splat
- Differentiable renderer → Reconstruction losses: images and visibility
- Reconstruction losses → Training CLI: gradients
- Training CLI → Model artifacts: checkpoints
- Model artifacts → Render CLI: load checkpoint
- Render CLI → Rendered predictions: write renders
- Rendered predictions → Metrics CLI: predictions
- Dynamic scene data → Metrics CLI: reference images
- Batch evaluation → Render CLI: orchestrates
- Batch evaluation → Metrics CLI: orchestrates