How microsoft/trellis works
Project Type and Purpose
• This is a full-stack machine-learning library and framework for 3D asset generation (text-to-3D, image-to-3D).
• It provides both training (train.py + trellis/trainers) and inference (example scripts + app.py + trellis/pipelines) capabilities, plus dataset tooling and model configurations.
High-Level File-Structure Overview
• Top-level scripts:
• train.py – entry point for training
• example*.py, app.py – inference/demo drivers
• configs/ – JSON configs for VAE & flow models
• dataset_toolkits/ – data preparation & rendering utilities
• trellis/ – core library
• datasets/ – dataset loaders & preprocessors
• models/ – model architectures (SLAT, VAE, flow)
• modules/ – building-block layers (attention, sparse conv)
• trainers/ – training loops & logic
• pipelines/ – inference pipelines (image→3D, text→3D)
• renderers/ – renderers for different 3D formats
• representations/ – data structures (gaussians, meshes, radiance fields)
• utils/ – misc. utilities (IO, postprocessing, dist)
Key Architectural Patterns
• Modular design: core “modules” (attention, sparse conv, etc.) are composed into “models” (SLAT VAE, flow transformers).
• Clear separation of concerns:
• Models (trellis/models) define network architectures
• Trainers (trellis/trainers) orchestrate loss, optimization, distributed training
• Pipelines (trellis/pipelines) package model + sampling algorithm for inference
• Renderers/Representations convert latent outputs to viewable 3D formats
• Config-driven: all hyperparameters and model-checkpoint pointers live in configs/*.json.
Main Components and Interactions
A. User Interfaces
• CLI: train.py, example.py, example_text.py, example_variant.py
• Web demo: app.py, app_text.py (Gradio)
B. Configuration Layer (configs/)
• JSON files for VAE and flow settings
• Fed to both train.py and pipeline constructors
C. Data Layer
• dataset_toolkits/ (data download, blender script, render)
• trellis/datasets/ (PyTorch Dataset + collate + transforms)
D. Model Components (trellis/models + trellis/modules)
• Modules:
• trellis/modules/attention (full, windowed, serialized)
• trellis/modules/sparse (conv, linear, norm, transformer blocks)
• Models:
• structured_latent_vae (encoder + decoders for mesh/gaussian/rf)
• structured_latent_flow (Rectified Flow Transformers)
• sparse_structure_vae / sparse_structure_flow
• sparse_elastic_mixin (shared mixin layers)
E. Training Orchestrator (trellis/trainers)
• Base trainers: trellis/trainers/base.py
• VAE trainers: trellis/trainers/vae/…
• Flow-matching trainers: trellis/trainers/flow_matching/…
• Mixins for guidance, text/image conditioning
F. Inference Pipelines (trellis/pipelines)
• trellis_image_to_3d.py & trellis_text_to_3d.py
• Sampler submodules (classifier_free_guidance, flow_euler)
G. Renderers & Output Representations
• trellis/representations/ (gaussian, mesh, octree, radiance_field)
• trellis/renderers/ (mesh_renderer, octree_renderer, gaussian_render)
• postprocessing_utils to export PLY, GLB, video
H. Utilities & External Dependencies
• trellis/utils/ (data_utils, dist_utils, grad_clip_utils, etc.)
• External libs: PyTorch, CUDA, xFormers/flash-attn, spconv, diffoctreerast, gradio
Architectural Patterns & Design Principles
• Layered modularity: separate data ingestion, model definition, training, inference, rendering
• Config-first: no hard-coded hyperparameters; everything driven by JSON
• Mixin & inheritance: flow-matching trainers reuse common guidance logic via mixins