tencent-hunyuan/hunyuan3d-1

1. This repository is a two-stage 3D generation system. The first stage is a multi-view generator, and the second stage is a sparse-view reconstruction model that turns those views into a 3D mesh. It supports both text-to-3D and image-to-3D through a shared pipeline.

2. The top-level entry points are split by use case:

- main.py is the command-line inference entry for generating 3D assets.

- app.py is the Gradio demo/server entry point for interactive use.

- scripts/*.sh wrap common configurations for standard vs. lite, and for combined vs. step-by-step execution.

3. The orchestration layer lives in infer/. This is the main glue code that coordinates the pipeline stages:

- text_to_image.py for prompt-to-image generation when the input is text.

- image_to_views.py for producing the fixed multi-view set.

- removebg.py for preprocessing input images.

- views_to_mesh.py for reconstruction from multi-view images.

- gif_render.py for visualization/output rendering.

- utils.py for shared inference helpers.

This package is where the end-to-end data flow is assembled.

4. The generation pipeline is separated into standard and lite implementations under mvd/:

- hunyuan3d_mvd_std_pipeline.py

- hunyuan3d_mvd_lite_pipeline.py

These correspond to the multi-view diffusion stage and differ mainly in capacity and memory/runtime tradeoffs. They are the first major model boundary in the system.

5. The reconstruction model is isolated under svrm/. This is the second major stage and contains the feed-forward 3D reconstruction network:

- svrm/predictor.py is the high-level predictor interface.

- svrm/ldm/models/svrm.py and svrm/configs/svrm.yaml define the model and configuration.

- svrm/ldm/modules/translator/img_to_triplane.py converts image features into triplane-style latent 3D representations.

- svrm/ldm/modules/rendering_neus/ contains mesh/rasterization/rendering logic used during reconstruction and supervision.

6. The internal SVRM architecture is modular. It includes:

- a Dinov2-based image encoder under svrm/ldm/modules/encoders/dinov2/,

- attention and transformer utilities,

- a renderer / neural surface pipeline under rendering_neus,

- math, ray sampling, and ray marching utilities for volumetric or surface-based rendering.

This suggests the reconstruction stage is not just mesh decoding; it combines learned feature extraction with differentiable rendering components.

7. The infer/ pipeline uses a fixed camera layout. The README states the output views are generated at azimuths 0, 60, 120, 180, 240, 300 relative to the input view. That fixed view set is the bridge between the multi-view diffusion stage and the mesh reconstruction stage.

8. Text-to-3D has an extra front-end image synthesis step. The README and tree indicate the text pipeline uses a text-to-image model before multi-view generation. The repo does not ship the text-image backbone itself in this tree, but README points to Hunyuan-DiT checkpoints, so the system depends on an external pretrained text-to-image model for prompt conditioning.

9. Image-to-3D starts from an input image and optionally removes the background before view synthesis. That preprocessing boundary matters because the reconstruction quality depends on the object-centric crop/alpha handling before multi-view generation and mesh recovery.

10. Texture and baking are optional post-processing stages. The repo supports:

- texture mapping / texture shading inside the main pipeline,

- baking in third_party/, which depends on Dust3R and associated utilities.

Baking is explicitly separated because it introduces an external license and extra dependencies, and it is not required for basic mesh generation.

11. Rendering and export are separate from generation. The pipeline can produce GIFs and rendered previews after mesh creation, but those are output-side concerns rather than part of the core model. This keeps generation, reconstruction, and presentation distinct.

12. The third_party/ directory is a clear boundary for non-core integration code. It includes Dust3R-based matching/baking helpers, camera/image utilities, and mesh baking scripts. These components are only needed for the optional texture-baking workflow, not for the core text/image-to-3D path.

13. Operationally, the architecture is GPU-centric and checkpoint-driven. The user downloads pretrained weights into weights/ and then selects either the lite or standard pipeline, with an additional --save_memory mode for lower VRAM at the cost of slower execution. That means memory footprint is an important deployment boundary in this repo.

14. In diagram terms, the main flow is:

input prompt or image -> optional text-to-image / background cleanup -> multi-view generation (mvd) -> sparse-view reconstruction (svrm) -> optional texture mapping / baking (third_party) -> optional rendering/export.

The two model stages are the primary subsystems, and infer/ is the orchestration layer that connects them.