# cvg/zipsplat architecture

> Architecture diagram of the GitHub repository cvg/zipsplat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cvg/zipsplat
- Repository: https://github.com/cvg/zipsplat

## Overview

#### Purpose and package boundary
ZipSplat reconstructs 3D Gaussian scenes from one or more images in one feed-forward pass, optionally using camera priors. The repository separates a lightweight inference package, `zipsplat/`, from the CUDA-oriented training/evaluation framework, `splatfactory/`. Primary user entry points are `zipsplat/zipsplat.py`, `zipsplat/viewer.py`, `splatfactory/train.py`, and `splatfactory/eval/run.py`.

#### Standalone inference pipeline
`ZipSplat` accepts images/video-derived frames, normalizes them internally, runs a DA3-based visual backbone, converts compact scene tokens into Gaussian groups, and returns a renderable Gaussian container. Query sampling controlled by `compression` reduces the retained token set and therefore Gaussian count. Core paths: `zipsplat/zipsplat.py`, `zipsplat/backbone.py`, `zipsplat/predictor.py`.

#### Scene representation, camera, and rendering
The public scene boundary is the `Gaussians` object: it owns Gaussian attributes, rendering, and PLY export. Camera and pose abstractions isolate projection and world/view transforms from model prediction. Rendering depends on CUDA-capable gsplat; inference can use stock gsplat. Core paths: `zipsplat/gaussians.py`, `zipsplat/camera.py`, `zipsplat/pose.py`.

#### Interactive and offline outputs
The optional browser viewer drives reconstruction and exposes live compression/token-color visualization. Programmatic visualization supports turntable/video generation, while exported PLY is interoperable with external Gaussian-splat viewers. Core paths: `zipsplat/viewer.py`, `zipsplat/viz.py`, `zipsplat/utils.py`.

#### Training model composition
`splatfactory` provides the trainable counterpart: a ZipSplat network combines the DA3 encoder, Gaussian decoder/head, losses, metrics, and scene geometry utilities. Hydra configuration selects model/backbone/data/train recipes and supports overrides. Core paths: `splatfactory/models/networks/zipsplat.py`, `splatfactory/models/encoders/dav3_encoder.py`, `splatfactory/models/decoders/gaussian_head.py`.

#### Data preparation and streaming
Large multi-view datasets are converted into a common tar-sharded scene format containing images, cameras, poses, and added depth. Dataset implementations compose sources and select context/target views; depth supports scale normalization and depth/chamfer supervision. Conversion/download and depth extraction are distinct preprocessing stages. Core paths: `splatfactory/datasets/multi_view_dataset.py`, `splatfactory/datasets/composed.py`, `splatfactory/datasets/scripts/extract_depth.py`.

#### Distributed optimization and evaluation
Training is launched through `splatfactory.train`, coordinated by the trainer and distributed utilities, with checkpoints under experiment outputs. Evaluation loads a release or experiment checkpoint, applies fixed benchmark view indices, renders held-out views, and reports PSNR/SSIM/LPIPS. Core paths: `splatfactory/trainer.py`, `splatfactory/eval/eval_pipeline.py`, `splatfactory/eval/indices/`.

#### Runtime and external services
Inference requires Python 3.10+ and a CUDA GPU for gsplat rendering; pretrained weights are automatically retrieved from Hugging Face. Training uses Python 3.12 in the documented setup, CUDA/nvcc, PyTorch, gsplat, and fused-ssim; its gsplat fork exposes an activated-Gaussian signal used by geometry loss handling. Optional integrations include xFormers, TensorBoard, Weights & Biases, and DA3 weights/depth extraction. Core dependency manifests: `pyproject.toml`, `splatfactory/pyproject.toml`, `splatfactory/settings.py`.

Main entry points: [`zipsplat/gaussians.py`](https://github.com/cvg/zipsplat/blob/main/zipsplat/gaussians.py) (Gaussians), [`splatfactory/trainer.py`](https://github.com/cvg/zipsplat/blob/main/splatfactory/trainer.py) (Trainer & distributed runtime), [`zipsplat/zipsplat.py`](https://github.com/cvg/zipsplat/blob/main/zipsplat/zipsplat.py) (ZipSplat API), [`splatfactory/eval/eval_pipeline.py`](https://github.com/cvg/zipsplat/blob/main/splatfactory/eval/eval_pipeline.py) (Evaluation pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["ZipSplat inference"]
  node_input_frames["Images / video frames<br/>input"]
  node_zipsplat_api["ZipSplat API<br/>inference entry point<br/>[zipsplat.py]"]
  node_da3_backbone["DA3 visual backbone<br/>feature encoder<br/>[backbone.py]"]
  node_gaussian_predictor["Gaussian predictor<br/>token decoder<br/>[predictor.py]"]
  node_gaussians["Gaussians<br/>public scene container<br/>[gaussians.py]"]
  node_camera_pose["Camera &amp; pose<br/>geometry abstractions<br/>[camera.py]"]
  node_viewer["Browser viewer<br/>interactive entry point<br/>[viewer.py]"]
  node_offline_viz["Offline visualization<br/>turntable/video output<br/>[viz.py]"]
  node_ply["PLY export<br/>interoperable output<br/>[gaussians.py]"]
end

subgraph group_training["SplatFactory training"]
  node_train_entry["Training launch<br/>Hydra entry point<br/>[train.py]"]
  node_train_network["Trainable ZipSplat<br/>network composition<br/>[zipsplat.py]"]
  node_train_encoder["DA3 encoder<br/>trainable encoder<br/>[dav3_encoder.py]"]
  node_gaussian_head["Gaussian head<br/>trainable decoder<br/>[gaussian_head.py]"]
  node_trainer["Trainer &amp; distributed runtime<br/>optimization coordinator<br/>[trainer.py]"]
  node_checkpoints[("Experiment checkpoints<br/>model artifacts")]
  node_eval_pipeline["Evaluation pipeline<br/>benchmark evaluator<br/>[eval_pipeline.py]"]
  node_benchmark_indices["Benchmark view indices<br/>evaluation split"]
end

subgraph group_data["Data pipeline"]
  node_prepared_scenes[("Tar-sharded scenes<br/>dataset format")]
  node_dataset_composition["Dataset composition<br/>view sampler<br/>[composed.py]"]
  node_depth_extraction["Depth extraction<br/>preprocessing stage<br/>[extract_depth.py]"]
  node_multiview_dataset["Multi-view dataset<br/>training data loader"]
end

node_gsplat{{"gsplat CUDA renderer<br/>rendering dependency"}}
node_hf_weights[("Hugging Face weights<br/>model registry")]

node_input_frames -->|"input"| node_zipsplat_api
node_hf_weights -.->|"loads weights"| node_zipsplat_api
node_zipsplat_api -->|"normalized frames"| node_da3_backbone
node_da3_backbone -->|"scene tokens"| node_gaussian_predictor
node_gaussian_predictor -->|"Gaussian groups"| node_gaussians
node_camera_pose -->|"projection transforms"| node_gaussians
node_gaussians -->|"render"| node_gsplat
node_viewer -->|"interactive reconstruction"| node_zipsplat_api
node_gaussians -->|"turntable / video"| node_offline_viz
node_gaussians -->|"export"| node_ply
node_depth_extraction -->|"adds depth"| node_prepared_scenes
node_prepared_scenes -->|"scene shards"| node_dataset_composition
node_dataset_composition -->|"context / target views"| node_multiview_dataset
node_multiview_dataset -->|"training batches"| node_train_entry
node_train_entry -->|"Hydra-configured run"| node_trainer
node_trainer -->|"optimize"| node_train_network
node_train_network -->|"encodes views"| node_train_encoder
node_train_encoder -->|"features"| node_gaussian_head
node_gaussian_head -->|"Gaussian predictions and losses"| node_trainer
node_trainer -->|"writes"| node_checkpoints
node_checkpoints -->|"loads"| node_eval_pipeline
node_benchmark_indices -->|"fixed held-out views"| node_eval_pipeline
node_multiview_dataset -->|"benchmark scenes"| node_eval_pipeline
node_eval_pipeline -->|"renders held-out views"| node_gsplat

click node_zipsplat_api "https://github.com/cvg/zipsplat/blob/main/zipsplat/zipsplat.py"
click node_da3_backbone "https://github.com/cvg/zipsplat/blob/main/zipsplat/backbone.py"
click node_gaussian_predictor "https://github.com/cvg/zipsplat/blob/main/zipsplat/predictor.py"
click node_gaussians "https://github.com/cvg/zipsplat/blob/main/zipsplat/gaussians.py"
click node_camera_pose "https://github.com/cvg/zipsplat/blob/main/zipsplat/camera.py"
click node_viewer "https://github.com/cvg/zipsplat/blob/main/zipsplat/viewer.py"
click node_offline_viz "https://github.com/cvg/zipsplat/blob/main/zipsplat/viz.py"
click node_ply "https://github.com/cvg/zipsplat/blob/main/zipsplat/gaussians.py"
click node_dataset_composition "https://github.com/cvg/zipsplat/blob/main/splatfactory/datasets/composed.py"
click node_depth_extraction "https://github.com/cvg/zipsplat/blob/main/splatfactory/datasets/scripts/extract_depth.py"
click node_multiview_dataset "https://github.com/cvg/zipsplat/blob/main/splatfactory/datasets/multi_view_dataset.py"
click node_train_entry "https://github.com/cvg/zipsplat/blob/main/splatfactory/train.py"
click node_train_network "https://github.com/cvg/zipsplat/blob/main/splatfactory/models/networks/zipsplat.py"
click node_train_encoder "https://github.com/cvg/zipsplat/blob/main/splatfactory/models/encoders/dav3_encoder.py"
click node_gaussian_head "https://github.com/cvg/zipsplat/blob/main/splatfactory/models/decoders/gaussian_head.py"
click node_trainer "https://github.com/cvg/zipsplat/blob/main/splatfactory/trainer.py"
click node_eval_pipeline "https://github.com/cvg/zipsplat/blob/main/splatfactory/eval/eval_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_input_frames,node_zipsplat_api,node_da3_backbone,node_gaussian_predictor,node_gaussians,node_camera_pose,node_viewer,node_offline_viz,node_ply toneBlue
class node_train_entry,node_train_network,node_train_encoder,node_gaussian_head,node_trainer,node_checkpoints,node_eval_pipeline,node_benchmark_indices toneAmber
class node_prepared_scenes,node_dataset_composition,node_depth_extraction,node_multiview_dataset toneMint
class node_gsplat,node_hf_weights toneNeutral
```

## Components

### ZipSplat inference

Lightweight reconstruction package.

- **Images / video frames**: input
- [ZipSplat API](https://github.com/cvg/zipsplat/blob/main/zipsplat/zipsplat.py): inference entry point · Normalizes inputs and reconstructs a scene in one pass. · `zipsplat/zipsplat.py`
- [DA3 visual backbone](https://github.com/cvg/zipsplat/blob/main/zipsplat/backbone.py): feature encoder · `zipsplat/backbone.py`
- [Gaussian predictor](https://github.com/cvg/zipsplat/blob/main/zipsplat/predictor.py): token decoder · Compression sampling controls retained tokens and Gaussian count. · `zipsplat/predictor.py`
- [Gaussians](https://github.com/cvg/zipsplat/blob/main/zipsplat/gaussians.py): public scene container · Owns Gaussian attributes, rendering, and PLY export. · `zipsplat/gaussians.py`
- [Camera & pose](https://github.com/cvg/zipsplat/blob/main/zipsplat/camera.py): geometry abstractions · Isolates projection and world/view transforms. · `zipsplat/camera.py`
- [Browser viewer](https://github.com/cvg/zipsplat/blob/main/zipsplat/viewer.py): interactive entry point · Drives reconstruction with compression and token-color controls. · `zipsplat/viewer.py`
- [Offline visualization](https://github.com/cvg/zipsplat/blob/main/zipsplat/viz.py): turntable/video output · `zipsplat/viz.py`
- [PLY export](https://github.com/cvg/zipsplat/blob/main/zipsplat/gaussians.py): interoperable output · `zipsplat/gaussians.py`

### SplatFactory training

CUDA-oriented training and evaluation framework.

- [Training launch](https://github.com/cvg/zipsplat/blob/main/splatfactory/train.py): Hydra entry point · Configuration selects model, backbone, data, and training recipes. · `splatfactory/train.py`
- [Trainable ZipSplat](https://github.com/cvg/zipsplat/blob/main/splatfactory/models/networks/zipsplat.py): network composition · Combines DA3 encoding, Gaussian head, losses, metrics, and geometry utilities. · `splatfactory/models/networks/zipsplat.py`
- [DA3 encoder](https://github.com/cvg/zipsplat/blob/main/splatfactory/models/encoders/dav3_encoder.py): trainable encoder · `splatfactory/models/encoders/dav3_encoder.py`
- [Gaussian head](https://github.com/cvg/zipsplat/blob/main/splatfactory/models/decoders/gaussian_head.py): trainable decoder · `splatfactory/models/decoders/gaussian_head.py`
- [Trainer & distributed runtime](https://github.com/cvg/zipsplat/blob/main/splatfactory/trainer.py): optimization coordinator · Runs distributed optimization and writes experiment checkpoints. · `splatfactory/trainer.py`
- **Experiment checkpoints**: model artifacts
- [Evaluation pipeline](https://github.com/cvg/zipsplat/blob/main/splatfactory/eval/eval_pipeline.py): benchmark evaluator · Renders held-out views and reports PSNR, SSIM, and LPIPS. · `splatfactory/eval/eval_pipeline.py`
- **Benchmark view indices**: evaluation split

### Data pipeline

Multi-view scene preparation and streaming.

- **Tar-sharded scenes**: dataset format · Scenes contain images, cameras, poses, and depth.
- [Dataset composition](https://github.com/cvg/zipsplat/blob/main/splatfactory/datasets/composed.py): view sampler · Composes sources and selects context and target views. · `splatfactory/datasets/composed.py`
- [Depth extraction](https://github.com/cvg/zipsplat/blob/main/splatfactory/datasets/scripts/extract_depth.py): preprocessing stage · Adds depth for normalized depth and chamfer supervision. · `splatfactory/datasets/scripts/extract_depth.py`
- [Multi-view dataset](https://github.com/cvg/zipsplat/blob/main/splatfactory/datasets/multi_view_dataset.py): training data loader · `splatfactory/datasets/multi_view_dataset.py`

### Other components

- **gsplat CUDA renderer**: rendering dependency · Stock gsplat supports inference rendering.
- **Hugging Face weights**: model registry · Pretrained weights are retrieved automatically.

## Connections

- Images / video frames → ZipSplat API: input
- Hugging Face weights → ZipSplat API: loads weights
- ZipSplat API → DA3 visual backbone: normalized frames
- DA3 visual backbone → Gaussian predictor: scene tokens
- Gaussian predictor → Gaussians: Gaussian groups
- Camera & pose → Gaussians: projection transforms
- Gaussians → gsplat CUDA renderer: render
- Browser viewer → ZipSplat API: interactive reconstruction
- Gaussians → Offline visualization: turntable / video
- Gaussians → PLY export: export
- Depth extraction → Tar-sharded scenes: adds depth
- Tar-sharded scenes → Dataset composition: scene shards
- Dataset composition → Multi-view dataset: context / target views
- Multi-view dataset → Training launch: training batches
- Training launch → Trainer & distributed runtime: Hydra-configured run
- Trainer & distributed runtime → Trainable ZipSplat: optimize
- Trainable ZipSplat → DA3 encoder: encodes views
- DA3 encoder → Gaussian head: features
- Gaussian head → Trainer & distributed runtime: Gaussian predictions and losses
- Trainer & distributed runtime → Experiment checkpoints: writes
- Experiment checkpoints → Evaluation pipeline: loads
- Benchmark view indices → Evaluation pipeline: fixed held-out views
- Multi-view dataset → Evaluation pipeline: benchmark scenes
- Evaluation pipeline → gsplat CUDA renderer: renders held-out views