cvg/zipsplat

Purpose and package boundary

ZipSplat reconstructs 3D Gaussian scenes from one or more images in one feed-forward pass, optionally using camera priors. The repository separates a lightweight inference package, zipsplat/, from the CUDA-oriented training/evaluation framework, splatfactory/. Primary user entry points are zipsplat/zipsplat.py, zipsplat/viewer.py, splatfactory/train.py, and splatfactory/eval/run.py.

Standalone inference pipeline

ZipSplat accepts images/video-derived frames, normalizes them internally, runs a DA3-based visual backbone, converts compact scene tokens into Gaussian groups, and returns a renderable Gaussian container. Query sampling controlled by compression reduces the retained token set and therefore Gaussian count. Core paths: zipsplat/zipsplat.py, zipsplat/backbone.py, zipsplat/predictor.py.

Scene representation, camera, and rendering

The public scene boundary is the Gaussians object: it owns Gaussian attributes, rendering, and PLY export. Camera and pose abstractions isolate projection and world/view transforms from model prediction. Rendering depends on CUDA-capable gsplat; inference can use stock gsplat. Core paths: zipsplat/gaussians.py, zipsplat/camera.py, zipsplat/pose.py.

Interactive and offline outputs

The optional browser viewer drives reconstruction and exposes live compression/token-color visualization. Programmatic visualization supports turntable/video generation, while exported PLY is interoperable with external Gaussian-splat viewers. Core paths: zipsplat/viewer.py, zipsplat/viz.py, zipsplat/utils.py.

Training model composition

splatfactory provides the trainable counterpart: a ZipSplat network combines the DA3 encoder, Gaussian decoder/head, losses, metrics, and scene geometry utilities. Hydra configuration selects model/backbone/data/train recipes and supports overrides. Core paths: splatfactory/models/networks/zipsplat.py, splatfactory/models/encoders/dav3_encoder.py, splatfactory/models/decoders/gaussian_head.py.

Data preparation and streaming

Large multi-view datasets are converted into a common tar-sharded scene format containing images, cameras, poses, and added depth. Dataset implementations compose sources and select context/target views; depth supports scale normalization and depth/chamfer supervision. Conversion/download and depth extraction are distinct preprocessing stages. Core paths: splatfactory/datasets/multi_view_dataset.py, splatfactory/datasets/composed.py, splatfactory/datasets/scripts/extract_depth.py.

Distributed optimization and evaluation

Training is launched through splatfactory.train, coordinated by the trainer and distributed utilities, with checkpoints under experiment outputs. Evaluation loads a release or experiment checkpoint, applies fixed benchmark view indices, renders held-out views, and reports PSNR/SSIM/LPIPS. Core paths: splatfactory/trainer.py, splatfactory/eval/eval_pipeline.py, splatfactory/eval/indices/.

Runtime and external services

Inference requires Python 3.10+ and a CUDA GPU for gsplat rendering; pretrained weights are automatically retrieved from Hugging Face. Training uses Python 3.12 in the documented setup, CUDA/nvcc, PyTorch, gsplat, and fused-ssim; its gsplat fork exposes an activated-Gaussian signal used by geometry loss handling. Optional integrations include xFormers, TensorBoard, Weights & Biases, and DA3 weights/depth extraction. Core dependency manifests: pyproject.toml, splatfactory/pyproject.toml, splatfactory/settings.py.