How nicolasvonluetzow/gaussiangpt works
GaussianGPT is a pipeline for autoregressive 3D Gaussian scene generation. Scene datasets feed a sparse VQ-VAE, which compresses Gaussian primitives into discrete latents; tokenization serializes those latents for causal-transformer training. At inference, the GPT samples token sequences, optionally conditioned for completion or spatial extension, and the frozen VQ-VAE decodes them into Gaussian scenes for rendering. The graph combines README-documented workflow with sampled model and data excerpts; some subsystem links rely on the README rather than inspected entry-point source.
Main entry points: generate_scene.py Scene generation · gaussian_vqvae.py Gaussian VQ-VAE · tokenize_dataset.py Dataset tokenization · gaussian_gpt.py Gaussian GPT
Components 22
Scene data
- Scene datasets
data/common.py - VFront data
data/vfront_dataset.py - PhotoShape data
data/photoshape_dataset.py
Gaussian compression
- VQ-VAE training
train_ae.py - Gaussian VQ-VAE
model/gaussian_vqvae.py - Sparse CNN
model/vqvae/cnn/configurable_vqvae.py - Gaussian formats
conf/internal_representations.py
Token modeling
- Dataset tokenization
tokenize_dataset.py - Spatial serialization
serialization/hilbert.py - Token streams
data/common.py - GPT training
train_gpt.py - Gaussian GPT
model/gaussian_gpt.py - Causal transformer
model/gpt/nanochat_gpt.py - Sampling engine
model/gpt/engine.py
Scene generation
- Scene generation
generate_scene.py - Completion and tiling
complete_chunks.py - Chunk generation
generate_chunks.py - Scene decoding
decode_scene.py - Scene rendering
utils/render.py - Gaussian scenes
Other components
- Research user
- gsplat renderer
Connections 26
- Research user → VQ-VAE training: trains
- Scene datasets → VQ-VAE training: provides scenes
- VFront data → Scene datasets: specializes
- PhotoShape data → Scene datasets: specializes
- VQ-VAE training → Gaussian VQ-VAE: optimizes
- Gaussian VQ-VAE → Sparse CNN: uses
- Gaussian VQ-VAE → Gaussian formats: converts features
- Research user → Dataset tokenization: tokenizes dataset
- Dataset tokenization → Gaussian VQ-VAE: encodes scenes
- Dataset tokenization → Spatial serialization: orders latents
- Dataset tokenization → Token streams: writes tokens
- Token streams → GPT training: provides sequences
- GPT training → Gaussian GPT: trains
- Gaussian GPT → Causal transformer: wraps
- Research user → Scene generation: requests scenes
- Scene generation → Gaussian GPT: samples tokens
- Gaussian GPT → Sampling engine: generates sequences
- Scene generation → Gaussian VQ-VAE: decodes tokens
- Scene generation → Gaussian scenes: produces
- Completion and tiling → Scene generation: conditions generation
- Chunk generation → Scene generation: generates chunks
- Scene decoding → Gaussian VQ-VAE: decodes tokens
- Scene decoding → Gaussian scenes: produces
- Scene rendering → gsplat renderer: renders with
- Scene rendering → Gaussian scenes: renders
- Scene generation → Scene rendering: renders output