How lidan233/caddreamer works
Project Type and Purpose
• CADDreamer is a research-grade, full-stack machine-learning and CAD reconstruction pipeline. Its goal is to transform a single RGB image into a compact, editable CAD B-rep via three major stages: (1) multi-view normal map diffusion, (2) neural 3D reconstruction (NEUS), and (3) CAD primitive segmentation and fitting (using RANSAC and FreeCAD/OpenCascade).
File Structure Overview
• Top-level:
• README.md, requirements.txt, setup.sh – documentation and environment setup
• configs/ – YAML files controlling training and test configurations
• cached_output/ – generated intermediate and final outputs
• test_*.py scripts – CLI entry points for each stage
• automl/, instant_nsr_pl/, mvdiffusion/, neus/, pyransac/, utils/, sam_pt/ – main code modules
• Key components:
• mvdiffusion/ – implements the multi-view diffusion model (PyTorch UNet, transformer blocks)
• instant_nsr_pl/ – PyTorch Lightning wrappers for neural surface reconstruction (NEUS)
• neus/ – pure-Python implementation of NEUS inference, surface fitting, primitive intersection/stitching
• automl/ – utility code for AutoencoderKL (latent diffusion VAE) fine-tuning
• utils/ – shared logging, I/O, visualization, callbacks
Architectural Patterns and Design Principles
• Modular, stage-based pipeline orchestrated by Python CLI scripts.
• Separation of concerns:
• Diffusion Stage (mvdiffusion + automl/ckpts)
• Reconstruction Stage (instant_nsr_pl + neus)
• CAD Fitting Stage (neus + pyransac + FreeCAD/OpenCascade)
• Config-driven via YAML files in configs/.
• Hybrid Python/C++ integration: heavy ML in Python; performance-critical RANSAC in C++ (pyransac + pybind11).
Main Components and Interactions
• Input: single-view RGB image
→ Feature Extraction & Diffusion (mvdiffusion.pipeline_mvdiffusion_image, AutoencoderKL)
→ Generates multi-view normal semantic maps
→ NEUS Reconstruction (instant_nsr_pl.systems + neus.models)
→ Produces a 3D mesh (point cloud + normals → implicit field → surface)
→ Segmentation & Primitive Detection (neus.fit_surfaces + pyransac)
→ Identifies and fits CAD primitives to mesh
→ Intersection & Stitching (neus.intersectionocc + FreeCAD/OpenCascade)
→ Generates final CAD STEP file