# vlabsic/trellis architecture

> Architecture diagram of the GitHub repository vlabsic/trellis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vlabsic/trellis
- Repository: https://github.com/vlabsic/trellis

## Overview

##### Project type and purpose
- This is a Python-based machine-learning library and demo application (“Trellis”) for generating 3D assets from text or image prompts.
- It provides CLI examples (`example.py`), a web demo (`app.py` built on Gradio), data-preparation toolkits (`dataset_toolkits`), core pipelines (`trellis/pipelines`), model definitions (`trellis/models`), representations and renderers, plus an optional native CUDA extension (`extensions/vox2seq`).

##### File structure
- Top level:
- `app.py`, `example.py`, `example_multi_image.py` → user-facing entry points (web and CLI).
- `dataset_toolkits/` → scripts and modules for downloading, encoding, rendering, and preprocessing 3D data for model training.
- `extensions/vox2seq/` → custom CUDA/PyTorch extension for voxel-to-sequence operations.
- `trellis/` → the core library, subdivided into:
– `models/` (SLAT backbones, flows, VAEs)
– `modules/` (attention, sparse/transformer building blocks)
– `pipelines/` (end‐to‐end image/text-to-3D generation)
– `renderers/` (mesh, Gaussian, octree renderers)
– `representations/` (internal data formats: Gaussian, mesh, octree, radiance field)
– `utils/` (general utilities, rendering helpers, randomness, postprocessing).
- Config/build: `setup.sh`, `setup.py` under the extension, `setup.sh` at top level to bootstrap conda env and compile submodules.

##### Insights from the README
- Central component: `TrellisImageTo3DPipeline` (and forthcoming `TrellisTextTo3DPipeline`), which orchestrates:
a. Input encoding (text or image)
b. Structured latent sampling (Rectified Flow Transformers)
c. Decoding to multiple output formats (Gaussians, radiance fields, meshes)
d. Rendering utilities and file exports (GLB, PLY, videos)
- External dependencies: PyTorch (+ CUDA 11.8/12.2), flash-attn or xformers, spconv, nvdiffrast, Kaolin, etc.

##### Elements
- Blue for Interfaces (CLI, Web)
- Green for Core Pipeline & Orchestration
- Orange for Model Backbones & Modules
- Purple for Representation formats & Renderers
- Gray for Utilities & Toolkits
- Red outline for External Dependencies

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces
    subgraph "User Interfaces"
        direction TB
        UI_Web["Web Demo (Gradio)"]:::ui
        UI_CLI1["CLI Example (Single Image)"]:::ui
        UI_CLI2["CLI Example (Multi-Image)"]:::ui
    end

    %% Core Pipeline
    subgraph "Core Pipeline Orchestrator"
        direction TB
        Pipeline_Base["base.py"]:::pipeline
        Pipeline_Image3D["trellis_image_to_3d.py"]:::pipeline
        Pipeline_Samplers["samplers/"]:::pipeline
    end

    %% Model Backbones
    subgraph "Model Backbones"
        direction TB
        Model_SLFlow["structured_latent_flow.py"]:::model
        Model_SLVAE["structured_latent_vae/"]:::model
        Model_SparseFlow["sparse_structure_flow.py"]:::model
        Model_SparseVAE["sparse_structure_vae.py"]:::model
    end

    %% Building-Block Modules
    subgraph "Building-Block Modules"
        direction TB
        Mod_Attn["attention/"]:::module
        Mod_Sparse["sparse/"]:::module
        Mod_Transformer["transformer/"]:::module
        Mod_Norm["norm.py"]:::module
        Mod_Utils["utils.py"]:::module
    end

    %% Representation Formats
    subgraph "Representation Formats"
        direction TB
        Rep_Gaussian["gaussian/"]:::repr
        Rep_Mesh["mesh/"]:::repr
        Rep_Octree["octree/"]:::repr
        Rep_Radiance["radiance_field/"]:::repr
    end

    %% Renderers
    subgraph "Renderers"
        direction TB
        Rend_Gauss["gaussian_render.py"]:::renderer
        Rend_Mesh["mesh_renderer.py"]:::renderer
        Rend_Octree["octree_renderer.py"]:::renderer
        Rend_SH["sh_utils.py"]:::renderer
    end

    %% Postprocessing & Export Utilities
    subgraph "Utilities"
        direction TB
        Util_Post["postprocessing_utils.py"]:::util
        Util_Render["render_utils.py"]:::util
        Util_General["general_utils.py"]:::util
        Util_Rand["random_utils.py"]:::util
    end

    %% Data Preparation Toolkits
    subgraph "Data Preparation Toolkits"
        direction TB
        Toolkit["dataset_toolkits/"]:::toolkit
    end

    %% Native CUDA Extension
    subgraph "CUDA Extension"
        direction TB
        Extension["extensions/vox2seq/"]:::extension
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        Deps["PyTorch, CUDA, xformers, spconv, Kaolin, nvdiffrast"]:::deps
    end

    %% Connections
    UI_Web -->|"input prompt"| Pipeline_Base
    UI_CLI1 -->|"input prompt"| Pipeline_Base
    UI_CLI2 -->|"input prompt"| Pipeline_Base

    Pipeline_Base -->|"uses ImageTo3DPipeline"| Pipeline_Image3D
    Pipeline_Image3D -->|"calls samplers"| Pipeline_Samplers

    Pipeline_Image3D -->|"invokes flows/VAEs"| Model_SLFlow
    Pipeline_Image3D -->|"invokes flows/VAEs"| Model_SparseFlow
    Pipeline_Image3D -->|"invokes VAEs"| Model_SLVAE
    Pipeline_Image3D -->|"invokes VAEs"| Model_SparseVAE

    Model_SLFlow -->|"uses attention modules"| Mod_Attn
    Model_SparseFlow -->|"uses sparse modules"| Mod_Sparse
    Model_SLVAE -->|"uses transformer modules"| Mod_Transformer

    Mod_Attn -->|"norm/utils"| Mod_Norm
    Mod_Sparse -->|"util funcs"| Mod_Utils

    Model_SLVAE -->|"outputs latents"| Rep_Gaussian
    Model_SLVAE -->|"outputs latents"| Rep_Mesh
    Model_SLVAE -->|"outputs latents"| Rep_Octree
    Model_SLVAE -->|"outputs latents"| Rep_Radiance

    Rep_Gaussian -->|"to renderer"| Rend_Gauss
    Rep_Mesh -->|"to renderer"| Rend_Mesh
    Rep_Octree -->|"to renderer"| Rend_Octree
    Rep_Radiance -->|"to sh utils"| Rend_SH

    Rend_Gauss -->|"rendered frames"| Util_Render
    Rend_Mesh -->|"rendered frames"| Util_Render
    Rend_Octree -->|"rendered frames"| Util_Render
    Rend_SH -->|"rendered frames"| Util_Render

    Util_Render -->|"export files"| Util_Post
    Util_Post -->|"save GLB/PLY"| UI_Web

    Toolkit -->|"prepare training data"| Pipeline_Base

    Extension -->|"hooks into pipeline/modules"| Pipeline_Base
    Extension -->|"hooks into pipeline/modules"| Mod_Sparse

    Deps --> Pipeline_Base
    Deps --> Model_SLFlow
    Deps --> Model_SLVAE
    Deps --> Mod_Attn

    %% Click Events
    click UI_Web "https://github.com/vlabsic/trellis/blob/main/app.py"
    click UI_CLI1 "https://github.com/vlabsic/trellis/blob/main/example.py"
    click UI_CLI2 "https://github.com/vlabsic/trellis/blob/main/example_multi_image.py"

    click Pipeline_Base "https://github.com/vlabsic/trellis/blob/main/trellis/pipelines/base.py"
    click Pipeline_Image3D "https://github.com/vlabsic/trellis/blob/main/trellis/pipelines/trellis_image_to_3d.py"
    click Pipeline_Samplers "https://github.com/vlabsic/trellis/tree/main/trellis/pipelines/samplers/"

    click Model_SLFlow "https://github.com/vlabsic/trellis/blob/main/trellis/models/structured_latent_flow.py"
    click Model_SLVAE "https://github.com/vlabsic/trellis/blob/main/trellis/models/structured_latent_vae/base.py"
    click Model_SparseFlow "https://github.com/vlabsic/trellis/blob/main/trellis/models/sparse_structure_flow.py"
    click Model_SparseVAE "https://github.com/vlabsic/trellis/blob/main/trellis/models/sparse_structure_vae.py"

    click Mod_Attn "https://github.com/vlabsic/trellis/tree/main/trellis/modules/attention/"
    click Mod_Sparse "https://github.com/vlabsic/trellis/tree/main/trellis/modules/sparse/"
    click Mod_Transformer "https://github.com/vlabsic/trellis/tree/main/trellis/modules/transformer/"
    click Mod_Norm "https://github.com/vlabsic/trellis/blob/main/trellis/modules/norm.py"
    click Mod_Utils "https://github.com/vlabsic/trellis/blob/main/trellis/modules/utils.py"

    click Rep_Gaussian "https://github.com/vlabsic/trellis/tree/main/trellis/representations/gaussian/"
    click Rep_Mesh "https://github.com/vlabsic/trellis/tree/main/trellis/representations/mesh/"
    click Rep_Octree "https://github.com/vlabsic/trellis/tree/main/trellis/representations/octree/"
    click Rep_Radiance "https://github.com/vlabsic/trellis/blob/main/trellis/representations/radiance_field/strivec.py"

    click Rend_Gauss "https://github.com/vlabsic/trellis/blob/main/trellis/renderers/gaussian_render.py"
    click Rend_Mesh "https://github.com/vlabsic/trellis/blob/main/trellis/renderers/mesh_renderer.py"
    click Rend_Octree "https://github.com/vlabsic/trellis/blob/main/trellis/renderers/octree_renderer.py"
    click Rend_SH "https://github.com/vlabsic/trellis/blob/main/trellis/renderers/sh_utils.py"

    click Util_Post "https://github.com/vlabsic/trellis/blob/main/trellis/utils/postprocessing_utils.py"
    click Util_Render "https://github.com/vlabsic/trellis/blob/main/trellis/utils/render_utils.py"
    click Util_General "https://github.com/vlabsic/trellis/blob/main/trellis/utils/general_utils.py"
    click Util_Rand "https://github.com/vlabsic/trellis/blob/main/trellis/utils/random_utils.py"

    click Toolkit "https://github.com/vlabsic/trellis/tree/main/dataset_toolkits/"
    click Extension "https://github.com/vlabsic/trellis/tree/main/extensions/vox2seq/"

    %% Styles
    classDef ui fill:#ADD8E6,stroke:#1E90FF,color:#000;
    classDef pipeline fill:#98FB98,stroke:#228B22,color:#000;
    classDef model fill:#FFA500,stroke:#FF8C00,color:#000;
    classDef module fill:#FFD700,stroke:#DAA520,color:#000;
    classDef repr fill:#DDA0DD,stroke:#8A2BE2,color:#000;
    classDef renderer fill:#BA55D3,stroke:#9400D3,color:#000;
    classDef util fill:#D3D3D3,stroke:#696969,color:#000;
    classDef toolkit fill:#D3D3D3,stroke:#696969,color:#000;
    classDef extension fill:#F0E68C,stroke:#B8860B,color:#000;
    classDef deps fill:#FFC0CB,stroke:#FF0000,color:#000;
```
