How ziangcao0312/physx-3d works
Project Type and Purpose
• PhysX-3D is a research-grade machine-learning codebase for “Physical-Grounded 3D Asset Generation.” It integrates data-preprocessing toolkits, model definitions (VAE + diffusion/flow), training/inference pipelines, and rendering utilities.
• Core features include dataset annotation scripts (dataset_toolkits, tools), model architectures and modules under trellis, training entry point (train.py), inference example (example.py), and external‐dependency management (setup.sh, download_pretrain.sh).
High-Level Architectural Components
a. Data Layer
• Dataset Toolkit (dataset_toolkits/) – ingestion, preprocessing, metadata extraction, Blender integration for textures.
• External Datasets – PhysXNet (via Hugging Face), ShapeNet for textures.
b. Configuration & Environment
• Config files (configs/vae, configs/generation) – JSON experiment parameters.
• Setup scripts (setup.sh, download_pretrain.sh) – conda env, dependencies, pre-trained weights.
c. Core ML Library (Treillis Submodule)
• trellis/datasets/ – PyTorch Dataset classes, samplers.
• trellis/models/ – VAE, flow/diffusion architectures (encoders, decoders, mixins).
• trellis/modules/ – attention, sparse and dense building blocks.
• trellis/pipelines/ – end-to-end training & inference orchestration (samplers, text/image->3D).
• trellis/trainers/ – training loops for VAE and flow matching, guidance mixins.
• trellis/renderers/ & trellis/utils/ – mesh/octree/gaussian rendering, loss, post-processing.
d. Voxel2Seq Library (Optional Component)
• vox2seq/ – CUDA/C++ extensions for voxel encoding (Hilbert, Z-order), PyTorch bindings.
e. Entry Points
• train.py – unified training CLI that loads config, initializes trellis pipelines/trainers, dispatches to GPU/distributed.
• example.py – inference demo: loads pre-trained models, runs generation, invokes renderers.
Component Interactions
• Data Toolkit → Preprocessed dataset on disk → Trellis Dataset classes
• Config Loader (train.py) → Instantiates appropriate Trainer (VAE or Flow) → Builds Model & Modules → Batches of data from Dataset → GPU Training
• After training → Model checkpoints → Inference pipeline (example.py) → Sampler pipelines → Renderers → Exported 3D assets
• Tools scripts (tools/) integrate with dataset or serve as annotation/visualization utilities.
Architectural Patterns & Principles
• Modularization: clear separation of data, models, modules, pipelines, trainers, and utilities.
• Pipeline Pattern: trellis/pipelines orchestrates data→model→sampler→renderer.
• Configuration-Driven: experiments and hyperparameters in JSON, enabling reproducibility.
• Mixins & Inheritance: sharing behaviors (classifier-free guidance, text/image conditioning) across samplers and trainers.