# hustvl/4dgaussians architecture

> Architecture diagram of the GitHub repository hustvl/4dgaussians, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hustvl/4dgaussians
- Repository: https://github.com/hustvl/4dgaussians

## Overview

#### Purpose and execution surface
This is a PyTorch research pipeline for reconstructing and real-time rendering dynamic scenes as time-conditioned 4D Gaussian Splatting models. Primary CLI entry points are `train.py` (optimization), `render.py` (checkpoint rendering), `metrics.py` (evaluation), and `full_eval.py` (batch evaluation). Auxiliary model transformations are `export_perframe_3DGS.py` and `merge_many_4dgs.py`.

#### Dataset ingestion and scene normalization
Dataset dispatch converts D-NeRF, HyperNeRF, DyNeRF/Neural 3D Video, COLMAP, and multi-camera layouts into a common camera/time/point-cloud scene representation. Readers handle source-specific poses, images, frame time, and initialization geometry; camera utilities adapt these for training and rendering.
- `scene/dataset_readers.py`
- `scene/dataset.py`
- `scene/hyper_loader.py`
- `scene/neural_3D_dataset_NDC.py`
- `scene/multipleview_dataset.py`

#### Gaussian scene model
The learned state is a set of canonical 3D Gaussians with appearance and geometric attributes, initialized from SfM/dense point clouds where available. It owns optimization-facing parameters, persistence/checkpoint behavior, and Gaussian lifecycle operations such as densification/pruning implied by the training pipeline.
- `scene/gaussian_model.py`
- `scene/__init__.py`
- `utils/point_utils.py`

#### Time-dependent deformation field
Dynamic motion is represented separately from canonical Gaussians: a neural deformation module maps spatial position and timestamp to changes used at render time. Its feature representation is based on multi-resolution HexPlane/grid components, with regularization controlling the learned spatiotemporal field.
- `scene/deformation.py`
- `scene/hexplane.py`
- `scene/grid.py`
- `scene/regulation.py`

#### Differentiable rendering boundary
The renderer combines a camera, timestamp, canonical Gaussian model, and deformation field, then delegates projection/rasterization to the CUDA rasterizer. Rendered images and auxiliary visibility/radii outputs feed losses during training or are persisted during offline rendering. A socket-oriented GUI layer can expose rendering interactively.
- `gaussian_renderer/__init__.py`
- `gaussian_renderer/network_gui.py`
- `submodules/depth-diff-gaussian-rasterization`
- `submodules/simple-knn`

#### Training and experiment orchestration
`train.py` parses shared/model/optimization arguments, loads a `arguments/...` scene preset, constructs scene and model state, and runs coarse/fine optimization stages. It writes experiment artifacts under the requested model/output path and supports checkpoint restart. Image reconstruction and perceptual losses come from reusable utility modules and the embedded LPIPS implementation.
- `train.py`
- `arguments/__init__.py`
- `utils/loss_utils.py`
- `lpipsPyTorch/modules/lpips.py`

#### Data preparation, outputs, and evaluation
External preprocessing establishes camera calibration and point-cloud initialization: COLMAP is invoked through `colmap.sh`; format-specific scripts extract frames or convert layouts. Training checkpoints are consumed by rendering and metrics workflows; rendered predictions are compared against dataset imagery using quality metrics. The project targets Python 3.7, PyTorch/CUDA (documented environment: PyTorch 1.13.1 + CUDA 11.6), COLMAP, and optional Nerfstudio preprocessing.
- `colmap.sh`
- `scripts/preprocess_dynerf.py`
- `scripts/downsample_point.py`
- `render.py`
- `metrics.py`

Main entry points: [`train.py`](https://github.com/hustvl/4dgaussians/blob/master/train.py) (Training CLI), [`gaussian_renderer/__init__.py`](https://github.com/hustvl/4dgaussians/blob/master/gaussian_renderer/__init__.py) (Differentiable renderer), [`scene/dataset_readers.py`](https://github.com/hustvl/4dgaussians/blob/master/scene/dataset_readers.py) (Dataset readers), [`scene/deformation.py`](https://github.com/hustvl/4dgaussians/blob/master/scene/deformation.py) (Deformation field)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Preparation"]
  node_raw_data[("Dynamic scene data<br/>external dataset")]
  node_preprocessing["Preprocessing<br/>data preparation<br/>[colmap.sh]"]
end

subgraph group_scene["Scene &amp; Models"]
  node_dataset_readers["Dataset readers<br/>dataset dispatch<br/>[dataset_readers.py]"]
  node_scene_dataset["Scene dataset<br/>scene representation<br/>[dataset.py]"]
  node_camera_adapters["Camera adapters<br/>camera utilities<br/>[hyper_loader.py]"]
  node_gaussian_model[("Canonical Gaussians<br/>learned scene state<br/>[gaussian_model.py]")]
  node_deformation["Deformation field<br/>time-conditioned network<br/>[deformation.py]"]
  node_hexplane["HexPlane grids<br/>spatiotemporal features<br/>[hexplane.py]"]
  node_regularization["Field regularization<br/>regularizer<br/>[regulation.py]"]
end

subgraph group_runtime["Training &amp; Rendering"]
  node_train["Training CLI<br/>optimization entry point<br/>[train.py]"]
  node_arguments["Arguments &amp; presets<br/>configuration interface<br/>[__init__.py]"]
  node_renderer["Differentiable renderer<br/>rendering boundary<br/>[__init__.py]"]
  node_cuda_rasterizer{{"CUDA rasterizer<br/>native rasterization"}}
  node_losses["Reconstruction losses<br/>training objective<br/>[loss_utils.py]"]
end

subgraph group_evaluation["Outputs &amp; Evaluation"]
  node_artifacts[("Model artifacts<br/>checkpoints &amp; outputs")]
  node_render["Render CLI<br/>checkpoint rendering<br/>[render.py]"]
  node_predictions["Rendered predictions<br/>rendered images"]
  node_metrics["Metrics CLI<br/>evaluation entry point<br/>[metrics.py]"]
  node_full_eval["Batch evaluation<br/>evaluation orchestration<br/>[full_eval.py]"]
end

node_raw_data -->|"prepare"| node_preprocessing
node_preprocessing -->|"calibrated inputs"| node_dataset_readers
node_raw_data -->|"source layouts"| node_dataset_readers
node_dataset_readers -->|"common scene data"| node_scene_dataset
node_dataset_readers -->|"poses and frames"| node_camera_adapters
node_scene_dataset -->|"initial point cloud"| node_gaussian_model
node_hexplane -->|"features"| node_deformation
node_regularization -->|"constrains"| node_deformation
node_arguments -->|"optimization settings"| node_train
node_scene_dataset -->|"cameras and times"| node_train
node_train -->|"optimizes"| node_gaussian_model
node_train -->|"optimizes"| node_deformation
node_camera_adapters -->|"camera and timestamp"| node_renderer
node_gaussian_model -->|"canonical state"| node_renderer
node_deformation -->|"dynamic offsets"| node_renderer
node_renderer -->|"project and splat"| node_cuda_rasterizer
node_renderer -->|"images and visibility"| node_losses
node_losses -->|"gradients"| node_train
node_train -->|"checkpoints"| node_artifacts
node_artifacts -->|"load checkpoint"| node_render
node_render -->|"write renders"| node_predictions
node_predictions -->|"predictions"| node_metrics
node_raw_data -->|"reference images"| node_metrics
node_full_eval -->|"orchestrates"| node_render
node_full_eval -->|"orchestrates"| node_metrics

click node_preprocessing "https://github.com/hustvl/4dgaussians/blob/master/colmap.sh"
click node_dataset_readers "https://github.com/hustvl/4dgaussians/blob/master/scene/dataset_readers.py"
click node_scene_dataset "https://github.com/hustvl/4dgaussians/blob/master/scene/dataset.py"
click node_camera_adapters "https://github.com/hustvl/4dgaussians/blob/master/scene/hyper_loader.py"
click node_gaussian_model "https://github.com/hustvl/4dgaussians/blob/master/scene/gaussian_model.py"
click node_deformation "https://github.com/hustvl/4dgaussians/blob/master/scene/deformation.py"
click node_hexplane "https://github.com/hustvl/4dgaussians/blob/master/scene/hexplane.py"
click node_regularization "https://github.com/hustvl/4dgaussians/blob/master/scene/regulation.py"
click node_train "https://github.com/hustvl/4dgaussians/blob/master/train.py"
click node_arguments "https://github.com/hustvl/4dgaussians/blob/master/arguments/__init__.py"
click node_renderer "https://github.com/hustvl/4dgaussians/blob/master/gaussian_renderer/__init__.py"
click node_cuda_rasterizer "https://github.com/hustvl/4dgaussians/tree/master/submodules/depth-diff-gaussian-rasterization"
click node_losses "https://github.com/hustvl/4dgaussians/blob/master/utils/loss_utils.py"
click node_render "https://github.com/hustvl/4dgaussians/blob/master/render.py"
click node_metrics "https://github.com/hustvl/4dgaussians/blob/master/metrics.py"
click node_full_eval "https://github.com/hustvl/4dgaussians/blob/master/full_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_data,node_preprocessing toneBlue
class node_dataset_readers,node_scene_dataset,node_camera_adapters,node_gaussian_model,node_deformation,node_hexplane,node_regularization toneAmber
class node_train,node_arguments,node_renderer,node_cuda_rasterizer,node_losses toneMint
class node_artifacts,node_render,node_predictions,node_metrics,node_full_eval toneRose
```

## Components

### Preparation

- **Dynamic scene data**: external dataset · D-NeRF, HyperNeRF, DyNeRF, COLMAP, and multi-camera inputs.
- [Preprocessing](https://github.com/hustvl/4dgaussians/blob/master/colmap.sh): data preparation · Extracts frames, calibrates cameras, and prepares point-cloud initialization. · `colmap.sh`

### Scene & Models

- [Dataset readers](https://github.com/hustvl/4dgaussians/blob/master/scene/dataset_readers.py): dataset dispatch · Normalizes source layouts into cameras, timestamps, and scene geometry. · `scene/dataset_readers.py`
- [Scene dataset](https://github.com/hustvl/4dgaussians/blob/master/scene/dataset.py): scene representation · `scene/dataset.py`
- [Camera adapters](https://github.com/hustvl/4dgaussians/blob/master/scene/hyper_loader.py): camera utilities · Adapts source cameras for optimization and rendering. · `scene/hyper_loader.py`
- [Canonical Gaussians](https://github.com/hustvl/4dgaussians/blob/master/scene/gaussian_model.py): learned scene state · Owns Gaussian parameters, checkpoints, densification, and pruning. · `scene/gaussian_model.py`
- [Deformation field](https://github.com/hustvl/4dgaussians/blob/master/scene/deformation.py): time-conditioned network · Maps canonical positions and timestamps to dynamic Gaussian changes. · `scene/deformation.py`
- [HexPlane grids](https://github.com/hustvl/4dgaussians/blob/master/scene/hexplane.py): spatiotemporal features · Multi-resolution feature representation for deformation. · `scene/hexplane.py`
- [Field regularization](https://github.com/hustvl/4dgaussians/blob/master/scene/regulation.py): regularizer · `scene/regulation.py`

### Training & Rendering

- [Training CLI](https://github.com/hustvl/4dgaussians/blob/master/train.py): optimization entry point · Runs coarse and fine optimization and supports restart checkpoints. · `train.py`
- [Arguments & presets](https://github.com/hustvl/4dgaussians/blob/master/arguments/__init__.py): configuration interface · `arguments/__init__.py`
- [Differentiable renderer](https://github.com/hustvl/4dgaussians/blob/master/gaussian_renderer/__init__.py): rendering boundary · Combines camera, time, Gaussians, and deformation into render outputs. · `gaussian_renderer/__init__.py`
- [CUDA rasterizer](https://github.com/hustvl/4dgaussians/tree/master/submodules/depth-diff-gaussian-rasterization): native rasterization · `submodules/depth-diff-gaussian-rasterization`
- [Reconstruction losses](https://github.com/hustvl/4dgaussians/blob/master/utils/loss_utils.py): training objective · Image and perceptual losses used for optimization. · `utils/loss_utils.py`

### Outputs & Evaluation

- **Model artifacts**: checkpoints & outputs · Experiment outputs at the requested model path.
- [Render CLI](https://github.com/hustvl/4dgaussians/blob/master/render.py): checkpoint rendering · `render.py`
- **Rendered predictions**: rendered images
- [Metrics CLI](https://github.com/hustvl/4dgaussians/blob/master/metrics.py): evaluation entry point · Compares predictions with dataset imagery. · `metrics.py`
- [Batch evaluation](https://github.com/hustvl/4dgaussians/blob/master/full_eval.py): evaluation orchestration · `full_eval.py`

## Connections

- Dynamic scene data → Preprocessing: prepare
- Preprocessing → Dataset readers: calibrated inputs
- Dynamic scene data → Dataset readers: source layouts
- Dataset readers → Scene dataset: common scene data
- Dataset readers → Camera adapters: poses and frames
- Scene dataset → Canonical Gaussians: initial point cloud
- HexPlane grids → Deformation field: features
- Field regularization → Deformation field: constrains
- Arguments & presets → Training CLI: optimization settings
- Scene dataset → Training CLI: cameras and times
- Training CLI → Canonical Gaussians: optimizes
- Training CLI → Deformation field: optimizes
- Camera adapters → Differentiable renderer: camera and timestamp
- Canonical Gaussians → Differentiable renderer: canonical state
- Deformation field → Differentiable renderer: dynamic offsets
- Differentiable renderer → CUDA rasterizer: project and splat
- Differentiable renderer → Reconstruction losses: images and visibility
- Reconstruction losses → Training CLI: gradients
- Training CLI → Model artifacts: checkpoints
- Model artifacts → Render CLI: load checkpoint
- Render CLI → Rendered predictions: write renders
- Rendered predictions → Metrics CLI: predictions
- Dynamic scene data → Metrics CLI: reference images
- Batch evaluation → Render CLI: orchestrates
- Batch evaluation → Metrics CLI: orchestrates
