# nicolasvonluetzow/gaussiangpt architecture

> Architecture diagram of the GitHub repository nicolasvonluetzow/gaussiangpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nicolasvonluetzow/gaussiangpt
- Repository: https://github.com/nicolasvonluetzow/gaussiangpt

## Overview

GaussianGPT is a pipeline for autoregressive 3D Gaussian scene generation. Scene datasets feed a sparse VQ-VAE, which compresses Gaussian primitives into discrete latents; tokenization serializes those latents for causal-transformer training. At inference, the GPT samples token sequences, optionally conditioned for completion or spatial extension, and the frozen VQ-VAE decodes them into Gaussian scenes for rendering. The graph combines README-documented workflow with sampled model and data excerpts; some subsystem links rely on the README rather than inspected entry-point source.

Main entry points: [`generate_scene.py`](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/generate_scene.py) (Scene generation), [`model/gaussian_vqvae.py`](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gaussian_vqvae.py) (Gaussian VQ-VAE), [`tokenize_dataset.py`](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/tokenize_dataset.py) (Dataset tokenization), [`model/gaussian_gpt.py`](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gaussian_gpt.py) (Gaussian GPT)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Scene data"]
  node_scene_data["Scene datasets<br/>[common.py]"]
  node_vfront["VFront data<br/>[vfront_dataset.py]"]
  node_photoshape["PhotoShape data"]
end

subgraph group_compression["Gaussian compression"]
  node_ae_train["VQ-VAE training<br/>[train_ae.py]"]
  node_vqvae["Gaussian VQ-VAE<br/>[gaussian_vqvae.py]"]
  node_sparse_cnn["Sparse CNN"]
  node_representations["Gaussian formats"]
end

subgraph group_sequence["Token modeling"]
  node_tokenize["Dataset tokenization"]
  node_serialization["Spatial serialization<br/>[hilbert.py]"]
  node_token_data["Token streams<br/>[common.py]"]
  node_gpt_train["GPT training<br/>[train_gpt.py]"]
  node_gaussian_gpt["Gaussian GPT<br/>[gaussian_gpt.py]"]
  node_transformer["Causal transformer<br/>[nanochat_gpt.py]"]
  node_sampling["Sampling engine<br/>[engine.py]"]
end

subgraph group_generation["Scene generation"]
  node_generation["Scene generation<br/>[generate_scene.py]"]
  node_completion["Completion and tiling<br/>[complete_chunks.py]"]
  node_chunk_generation["Chunk generation<br/>[generate_chunks.py]"]
  node_decode["Scene decoding<br/>[decode_scene.py]"]
  node_render["Scene rendering<br/>[render.py]"]
  node_gaussian_output["Gaussian scenes"]
end

node_user(("Research user"))
node_gsplat["gsplat renderer"]

node_user -->|"trains"| node_ae_train
node_scene_data -->|"provides scenes"| node_ae_train
node_vfront -->|"specializes"| node_scene_data
node_photoshape -->|"specializes"| node_scene_data
node_ae_train -->|"optimizes"| node_vqvae
node_vqvae -->|"uses"| node_sparse_cnn
node_vqvae -->|"converts features"| node_representations
node_user -->|"tokenizes dataset"| node_tokenize
node_tokenize -->|"encodes scenes"| node_vqvae
node_tokenize -->|"orders latents"| node_serialization
node_tokenize -->|"writes tokens"| node_token_data
node_token_data -->|"provides sequences"| node_gpt_train
node_gpt_train -->|"trains"| node_gaussian_gpt
node_gaussian_gpt -->|"wraps"| node_transformer
node_user -->|"requests scenes"| node_generation
node_generation -->|"samples tokens"| node_gaussian_gpt
node_gaussian_gpt -->|"generates sequences"| node_sampling
node_generation -->|"decodes tokens"| node_vqvae
node_generation -->|"produces"| node_gaussian_output
node_completion -->|"conditions generation"| node_generation
node_chunk_generation -->|"generates chunks"| node_generation
node_decode -->|"decodes tokens"| node_vqvae
node_decode -->|"produces"| node_gaussian_output
node_render -.->|"renders with"| node_gsplat
node_render -->|"renders"| node_gaussian_output
node_generation -->|"renders output"| node_render

click node_scene_data "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/common.py"
click node_vfront "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/vfront_dataset.py"
click node_photoshape "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/photoshape_dataset.py"
click node_ae_train "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/train_ae.py"
click node_vqvae "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gaussian_vqvae.py"
click node_sparse_cnn "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/vqvae/cnn/configurable_vqvae.py"
click node_representations "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/conf/internal_representations.py"
click node_tokenize "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/tokenize_dataset.py"
click node_serialization "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/serialization/hilbert.py"
click node_token_data "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/common.py"
click node_gpt_train "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/train_gpt.py"
click node_gaussian_gpt "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gaussian_gpt.py"
click node_transformer "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gpt/nanochat_gpt.py"
click node_sampling "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gpt/engine.py"
click node_generation "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/generate_scene.py"
click node_completion "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/complete_chunks.py"
click node_chunk_generation "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/generate_chunks.py"
click node_decode "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/decode_scene.py"
click node_render "https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/utils/render.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scene_data,node_vfront,node_photoshape,node_user toneBlue
class node_ae_train,node_vqvae,node_sparse_cnn,node_representations toneAmber
class node_tokenize,node_serialization,node_token_data,node_gpt_train,node_gaussian_gpt,node_transformer,node_sampling toneMint
class node_generation,node_completion,node_chunk_generation,node_decode,node_render,node_gaussian_output toneRose
class node_gsplat toneIndigo
```

## Components

### Scene data

- [Scene datasets](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/common.py): `data/common.py`
- [VFront data](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/vfront_dataset.py): `data/vfront_dataset.py`
- [PhotoShape data](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/photoshape_dataset.py): `data/photoshape_dataset.py`

### Gaussian compression

- [VQ-VAE training](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/train_ae.py): `train_ae.py`
- [Gaussian VQ-VAE](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gaussian_vqvae.py): `model/gaussian_vqvae.py`
- [Sparse CNN](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/vqvae/cnn/configurable_vqvae.py): `model/vqvae/cnn/configurable_vqvae.py`
- [Gaussian formats](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/conf/internal_representations.py): `conf/internal_representations.py`

### Token modeling

- [Dataset tokenization](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/tokenize_dataset.py): `tokenize_dataset.py`
- [Spatial serialization](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/serialization/hilbert.py): `serialization/hilbert.py`
- [Token streams](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/data/common.py): `data/common.py`
- [GPT training](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/train_gpt.py): `train_gpt.py`
- [Gaussian GPT](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gaussian_gpt.py): `model/gaussian_gpt.py`
- [Causal transformer](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gpt/nanochat_gpt.py): `model/gpt/nanochat_gpt.py`
- [Sampling engine](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/model/gpt/engine.py): `model/gpt/engine.py`

### Scene generation

- [Scene generation](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/generate_scene.py): `generate_scene.py`
- [Completion and tiling](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/complete_chunks.py): `complete_chunks.py`
- [Chunk generation](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/generate_chunks.py): `generate_chunks.py`
- [Scene decoding](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/decode_scene.py): `decode_scene.py`
- [Scene rendering](https://github.com/nicolasvonluetzow/gaussiangpt/blob/main/utils/render.py): `utils/render.py`
- **Gaussian scenes**

### Other components

- **Research user**
- **gsplat renderer**

## Connections

- Research user → VQ-VAE training: trains
- Scene datasets → VQ-VAE training: provides scenes
- VFront data → Scene datasets: specializes
- PhotoShape data → Scene datasets: specializes
- VQ-VAE training → Gaussian VQ-VAE: optimizes
- Gaussian VQ-VAE → Sparse CNN: uses
- Gaussian VQ-VAE → Gaussian formats: converts features
- Research user → Dataset tokenization: tokenizes dataset
- Dataset tokenization → Gaussian VQ-VAE: encodes scenes
- Dataset tokenization → Spatial serialization: orders latents
- Dataset tokenization → Token streams: writes tokens
- Token streams → GPT training: provides sequences
- GPT training → Gaussian GPT: trains
- Gaussian GPT → Causal transformer: wraps
- Research user → Scene generation: requests scenes
- Scene generation → Gaussian GPT: samples tokens
- Gaussian GPT → Sampling engine: generates sequences
- Scene generation → Gaussian VQ-VAE: decodes tokens
- Scene generation → Gaussian scenes: produces
- Completion and tiling → Scene generation: conditions generation
- Chunk generation → Scene generation: generates chunks
- Scene decoding → Gaussian VQ-VAE: decodes tokens
- Scene decoding → Gaussian scenes: produces
- Scene rendering → gsplat renderer: renders with
- Scene rendering → Gaussian scenes: renders
- Scene generation → Scene rendering: renders output