How jayliu0313/shape-guided works
Project Type and Purpose
• This is a research-grade machine-learning pipeline for 3D anomaly detection and segmentation.
• It ingests RGB images + 3D point clouds (MvTec3D-AD), preprocesses data, trains a 3D “expert” network, builds a dual-memory bank, and runs inference to produce anomaly heatmaps.
Main Architectural Components
a. Data Layer
• External Dataset (MvTec3D-AD)
• Local storage (raw point clouds, RGB images)
b. Preprocessing & Patch Generation
• tools/preprocessing.py → background removal
• cut_patches.py → divides point clouds into local patches
c. 3D Expert Model Training
• train_3Dmodel.py
• core/model.py & core/pointnet2_utils.py → PointNet2-based feature extractor
• checkpoint/storage (best_ckpt/*.pth)
d. Dual-Memory Construction (Inference Prep)
• main.py reads trained checkpoint
• core/shape_guide_core.py builds two memories:
– RGB-guided memory
– SDF (signed distance function) memory
e. Inference & Visualization
• core/features.py, core/rgb_sdf_feature.py compute complementary heatmaps
• tools/visualize_loss.py, utils/visualize_util.py for plotting results
f. Utilities & Configuration
• utils/* modules for helper functions, data I/O, metrics (AUROC, AUPRO)
• Configuration.pickle / .txt files in checkpoint for hyperparams
Relationships & Data Flow
1) Dataset → 2) Preprocessing → 3) Patch Generation → 4) Model Training → 5) Memory Construction → 6) Inference → 7) Output Heatmaps
• That preprocessing and patch-cutting must run in sequence.
Architectural Patterns & Design Principles
• Modular pipeline: separate scripts for each stage
• Core library modules (core/, utils/) implement reusable logic
• Checkpointing for reproducibility & skipping expensive steps
• Dual-memory design: two parallel memory stores fused at inference
Technologies & Frameworks
• Python 3.6+
• PyTorch 1.7+, CUDA
• PointNet2 architecture (pointnet2_utils)
• NumPy, Open3D or custom point-cloud tools (implicitly in preprocessing)