# microsoft/trellis architecture

> Architecture diagram of the GitHub repository microsoft/trellis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/trellis
- Repository: https://github.com/microsoft/trellis

## Overview

##### Project Type and Purpose
- This is a full-stack machine-learning library and framework for 3D asset generation (text-to-3D, image-to-3D).
- It provides both training (train.py + trellis/trainers) and inference (example scripts + app.py + trellis/pipelines) capabilities, plus dataset tooling and model configurations.

##### High-Level File-Structure Overview
- Top-level scripts:
- train.py              – entry point for training
- example*.py, app.py   – inference/demo drivers
- configs/               – JSON configs for VAE & flow models
- dataset_toolkits/      – data preparation & rendering utilities
- trellis/               – core library
- datasets/             – dataset loaders & preprocessors
- models/               – model architectures (SLAT, VAE, flow)
- modules/              – building-block layers (attention, sparse conv)
- trainers/             – training loops & logic
- pipelines/            – inference pipelines (image→3D, text→3D)
- renderers/            – renderers for different 3D formats
- representations/      – data structures (gaussians, meshes, radiance fields)
- utils/                – misc. utilities (IO, postprocessing, dist)

##### Key Architectural Patterns
- Modular design: core “modules” (attention, sparse conv, etc.) are composed into “models” (SLAT VAE, flow transformers).
- Clear separation of concerns:
- **Models** (trellis/models) define network architectures
- **Trainers** (trellis/trainers) orchestrate loss, optimization, distributed training
- **Pipelines** (trellis/pipelines) package model + sampling algorithm for inference
- **Renderers/Representations** convert latent outputs to viewable 3D formats
- Config-driven: all hyperparameters and model-checkpoint pointers live in configs/*.json.

##### Main Components and Interactions

A. **User Interfaces**
- CLI: train.py, example.py, example_text.py, example_variant.py
- Web demo: app.py, app_text.py (Gradio)

B. **Configuration Layer (configs/)**
- JSON files for VAE and flow settings
- Fed to both train.py and pipeline constructors

C. **Data Layer**
- dataset_toolkits/                  (data download, blender script, render)
- trellis/datasets/                  (PyTorch Dataset + collate + transforms)

D. **Model Components** (trellis/models + trellis/modules)
- **Modules**:
- trellis/modules/attention (full, windowed, serialized)
- trellis/modules/sparse (conv, linear, norm, transformer blocks)
- **Models**:
- structured_latent_vae (encoder + decoders for mesh/gaussian/rf)
- structured_latent_flow (Rectified Flow Transformers)
- sparse_structure_vae / sparse_structure_flow
- sparse_elastic_mixin (shared mixin layers)

E. **Training Orchestrator** (trellis/trainers)
- Base trainers: trellis/trainers/base.py
- VAE trainers: trellis/trainers/vae/…
- Flow-matching trainers: trellis/trainers/flow_matching/…
- Mixins for guidance, text/image conditioning

F. **Inference Pipelines** (trellis/pipelines)
- trellis_image_to_3d.py & trellis_text_to_3d.py
- Sampler submodules (classifier_free_guidance, flow_euler)

G. **Renderers & Output Representations**
- trellis/representations/ (gaussian, mesh, octree, radiance_field)
- trellis/renderers/ (mesh_renderer, octree_renderer, gaussian_render)
- postprocessing_utils to export PLY, GLB, video

H. **Utilities & External Dependencies**
- trellis/utils/         (data_utils, dist_utils, grad_clip_utils, etc.)
- External libs: PyTorch, CUDA, xFormers/flash-attn, spconv, diffoctreerast, gradio

##### Architectural Patterns & Design Principles
- Layered modularity: separate data ingestion, model definition, training, inference, rendering
- Config-first: no hard-coded hyperparameters; everything driven by JSON
- Mixin & inheritance: flow-matching trainers reuse common guidance logic via mixins

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces
    subgraph "User Interfaces"
        UItrain["train.py"]:::frontend
        UIex1["example.py"]:::frontend
        UIex2["example_multi_image.py"]:::frontend
        UIex3["example_text.py"]:::frontend
        UIex4["example_variant.py"]:::frontend
        UIweb1["app.py"]:::frontend
        UIweb2["app_text.py"]:::frontend
    end

    %% Configuration Layer
    Configs["configs/"]:::config

    %% Data Layer
    subgraph "Data Layer"
        DTK["dataset_toolkits/"]:::data
        Datasets["trellis/datasets/"]:::data
    end

    %% Modules and Models
    subgraph "Building-Block Modules"
        ModAtt["trellis/modules/attention/"]:::core
        ModSparse["trellis/modules/sparse/"]:::core
        ModTrans["trellis/modules/transformer/"]:::core
        ModNorm["trellis/modules/norm.py"]:::core
        ModUtils["trellis/modules/utils.py"]:::core
    end

    subgraph "Core Model Architectures"
        ModelSLAT["structured_latent_vae/"]:::core
        ModelFlow["structured_latent_flow.py"]:::core
        ModelSSVAE["sparse_structure_vae.py"]:::core
        ModelSSFlow["sparse_structure_flow.py"]:::core
        ModelMixin["sparse_elastic_mixin.py"]:::core
    end

    %% Training Orchestrator
    subgraph "Training Orchestrator"
        TBase["base.py"]:::core
        TBasic["basic.py"]:::core
        TUtils["utils.py"]:::core
        TVAE["trellis/trainers/vae/"]:::core
        TFlow["trellis/trainers/flow_matching/"]:::core
    end

    %% Inference Pipelines
    subgraph "Inference Pipelines"
        PBase["base.py"]:::core
        PImage["trellis/pipelines/trellis_image_to_3d.py"]:::core
        PText["trellis/pipelines/trellis_text_to_3d.py"]:::core
        PSamplers["trellis/pipelines/samplers/"]:::core
    end

    %% Renderers & Representations
    subgraph "Renderers & Output"
        Reps["trellis/representations/"]:::core
        Renderers["trellis/renderers/"]:::core
    end

    %% Utilities & External
    Utils["trellis/utils/"]:::utils
    HF["Hugging Face Hub"]:::external

    %% Connections - UI to Configs, Data, Pipelines, Trainers
    UItrain -->|uses| Configs
    UItrain -->|loads data| Datasets
    UItrain -->|starts| TBase
    UIex1 -->|loads| Configs
    UIex2 -->|loads| Configs
    UIex3 -->|loads| Configs
    UIex4 -->|loads| Configs
    UIex1 -->|calls| PImage
    UIex2 -->|calls| PImage
    UIex3 -->|calls| PText
    UIex4 -->|calls| PText
    UIweb1 -->|serves| PImage
    UIweb2 -->|serves| PText

    %% Configs to Trainers & Pipelines
    Configs -->|feeds| TVAE
    Configs -->|feeds| TFlow
    Configs -->|feeds| PImage
    Configs -->|feeds| PText

    %% Data to Trainers & Pipelines
    DTK -->|prepares| Datasets
    Datasets -->|provides| TVAE
    Datasets -->|provides| TFlow
    Datasets -->|provides| PSamplers

    %% Modules to Models
    ModAtt -->|used by| ModelSLAT
    ModSparse -->|used by| ModelSSVAE
    ModSparse -->|used by| ModelSSFlow
    ModTrans -->|used by| ModelFlow
    ModNorm -->|used by| ModelSLAT
    ModUtils -->|used by| ModelSLAT

    %% Models to Trainers & Pipelines
    ModelSLAT -->|trained in| TVAE
    ModelSSVAE -->|trained in| TVAE
    ModelFlow -->|trained in| TFlow
    ModelSSFlow -->|trained in| TFlow
    ModelMixin -->|mixes into| TVAE
    ModelMixin -->|mixes into| TFlow
    ModelSLAT -->|loaded by| PImage
    ModelFlow -->|loaded by| PText

    %% Trainers produce checkpoints
    TVAE -->|writes| Checkpoints["Checkpoints"]:::external
    TFlow -->|writes| Checkpoints

    %% Pipelines to Representations & Renderers
    PImage -->|outputs| Reps
    PText -->|outputs| Reps
    Reps -->|rendered by| Renderers
    Renderers -->|saves| UIweb1
    Renderers -->|saves| UIweb2

    %% Utilities support all
    Utils --> TVAE
    Utils --> TFlow
    Utils --> PImage
    Utils --> PText
    Utils --> ModelSLAT

    %% External Dependencies
    HF -->|provides weights| ModelSLAT
    HF -->|provides weights| ModelFlow

    %% Click Events
    click UItrain "https://github.com/microsoft/trellis/blob/main/train.py"
    click UIex1 "https://github.com/microsoft/trellis/blob/main/example.py"
    click UIex2 "https://github.com/microsoft/trellis/blob/main/example_multi_image.py"
    click UIex3 "https://github.com/microsoft/trellis/blob/main/example_text.py"
    click UIex4 "https://github.com/microsoft/trellis/blob/main/example_variant.py"
    click UIweb1 "https://github.com/microsoft/trellis/blob/main/app.py"
    click UIweb2 "https://github.com/microsoft/trellis/blob/main/app_text.py"
    click Configs "https://github.com/microsoft/trellis/tree/main/configs/"
    click DTK "https://github.com/microsoft/trellis/tree/main/dataset_toolkits/"
    click Datasets "https://github.com/microsoft/trellis/tree/main/trellis/datasets/"
    click ModAtt "https://github.com/microsoft/trellis/tree/main/trellis/modules/attention/"
    click ModSparse "https://github.com/microsoft/trellis/tree/main/trellis/modules/sparse/"
    click ModTrans "https://github.com/microsoft/trellis/tree/main/trellis/modules/transformer/"
    click ModNorm "https://github.com/microsoft/trellis/blob/main/trellis/modules/norm.py"
    click ModUtils "https://github.com/microsoft/trellis/blob/main/trellis/modules/utils.py"
    click ModelSLAT "https://github.com/microsoft/trellis/tree/main/trellis/models/structured_latent_vae/"
    click ModelFlow "https://github.com/microsoft/trellis/blob/main/trellis/models/structured_latent_flow.py"
    click ModelSSVAE "https://github.com/microsoft/trellis/blob/main/trellis/models/sparse_structure_vae.py"
    click ModelSSFlow "https://github.com/microsoft/trellis/blob/main/trellis/models/sparse_structure_flow.py"
    click ModelMixin "https://github.com/microsoft/trellis/blob/main/trellis/models/sparse_elastic_mixin.py"
    click TBase "https://github.com/microsoft/trellis/blob/main/trellis/trainers/base.py"
    click TBasic "https://github.com/microsoft/trellis/blob/main/trellis/trainers/basic.py"
    click TUtils "https://github.com/microsoft/trellis/blob/main/trellis/trainers/utils.py"
    click TVAE "https://github.com/microsoft/trellis/tree/main/trellis/trainers/vae/"
    click TFlow "https://github.com/microsoft/trellis/tree/main/trellis/trainers/flow_matching/"
    click PBase "https://github.com/microsoft/trellis/blob/main/trellis/pipelines/base.py"
    click PImage "https://github.com/microsoft/trellis/blob/main/trellis/pipelines/trellis_image_to_3d.py"
    click PText "https://github.com/microsoft/trellis/blob/main/trellis/pipelines/trellis_text_to_3d.py"
    click PSamplers "https://github.com/microsoft/trellis/tree/main/trellis/pipelines/samplers/"
    click Reps "https://github.com/microsoft/trellis/tree/main/trellis/representations/"
    click Renderers "https://github.com/microsoft/trellis/tree/main/trellis/renderers/"
    click Utils "https://github.com/microsoft/trellis/tree/main/trellis/utils/"

    %% Styles
    classDef frontend fill:#D39BE6,color:#000,stroke:#333,stroke-width:1px
    classDef data fill:#9BE6D3,color:#000,stroke:#333,stroke-width:1px
    classDef config fill:#F6C87B,color:#000,stroke:#333,stroke-width:1px
    classDef core fill:#89CFF0,color:#000,stroke:#333,stroke-width:1px
    classDef utils fill:#CCCCCC,color:#000,stroke:#333,stroke-width:1px
    classDef external fill:#EEEEEE,color:#000,stroke:#333,stroke-width:1px
```
