How vlabsic/trellis works
Project type and purpose
• This is a Python-based machine-learning library and demo application (“Trellis”) for generating 3D assets from text or image prompts.
• It provides CLI examples (example.py), a web demo (app.py built on Gradio), data-preparation toolkits (dataset_toolkits), core pipelines (trellis/pipelines), model definitions (trellis/models), representations and renderers, plus an optional native CUDA extension (extensions/vox2seq).
File structure
• Top level:
• app.py, example.py, example_multi_image.py → user-facing entry points (web and CLI).
• dataset_toolkits/ → scripts and modules for downloading, encoding, rendering, and preprocessing 3D data for model training.
• extensions/vox2seq/ → custom CUDA/PyTorch extension for voxel-to-sequence operations.
• trellis/ → the core library, subdivided into:
– models/ (SLAT backbones, flows, VAEs)
– modules/ (attention, sparse/transformer building blocks)
– pipelines/ (end‐to‐end image/text-to-3D generation)
– renderers/ (mesh, Gaussian, octree renderers)
– representations/ (internal data formats: Gaussian, mesh, octree, radiance field)
– utils/ (general utilities, rendering helpers, randomness, postprocessing).
• Config/build: setup.sh, setup.py under the extension, setup.sh at top level to bootstrap conda env and compile submodules.
Insights from the README
• Central component: TrellisImageTo3DPipeline (and forthcoming TrellisTextTo3DPipeline), which orchestrates:
a. Input encoding (text or image)
b. Structured latent sampling (Rectified Flow Transformers)
c. Decoding to multiple output formats (Gaussians, radiance fields, meshes)
d. Rendering utilities and file exports (GLB, PLY, videos)
• External dependencies: PyTorch (+ CUDA 11.8/12.2), flash-attn or xformers, spconv, nvdiffrast, Kaolin, etc.
Elements
• Blue for Interfaces (CLI, Web)
• Green for Core Pipeline & Orchestration
• Orange for Model Backbones & Modules
• Purple for Representation formats & Renderers
• Gray for Utilities & Toolkits
• Red outline for External Dependencies