How zju3dv/neuralrecon works
Project type and purpose
• NeuralRecon is a research‐code full-stack machine-learning pipeline for real-time coherent 3D reconstruction from monocular video.
• It includes data preparation, model training, inference, evaluation, and a live demo (including ARKit support).
File structure
• Top-level scripts (main.py, demo.py, train.sh, test.sh, run_demo_batch.sh) drive each stage.
• config/: YAML/Python configs for train/val/test/demo.
• datasets/: data-loading, sampling, transforms for ScanNet and ARKit.
• models/: network definitions (backbone, GRU fusion, NeuCon network, full recon head).
• ops/: CUDA/PyTorch custom ops (back-projection, TSDF grid gen, TorchSparse helpers).
• tools/: preprocessing (Generate GT TSDFs), postprocessing (evaluation, render, metrics, ARKit utilities).
• assets/: demo GIFs, visual examples.
• environment.yaml, requirements.txt: dependency management.
• checkpoints/, results/: where pretrained weights and outputs live.
README insights
• Describes training phases (fragment‐level then GRU fusion).
• Inference pipeline produces meshes then evaluation via pyrender.
• Real-time demo uses camera streams + live fusion.
• Data prep scripts generate ground-truth TSDF volumes using tools/tsdf_fusion.
a. Main components
• Data Sources
– ScanNet raw images + poses
– Custom ARKit video + poses
• Data Preparation
– tools/tsdf_fusion/generate_gt.py → ground-truth TSDFs
– datasets/…: samplers, transforms → PyTorch DataLoader
• Configuration Manager
– config/*.yaml/python → global hyperparameters & paths
• Training Pipeline
– Entry: train.sh → torch.distributed launch → main.py (train mode)
– Models: models/backbone → models/neucon_network & models/gru_fusion → models/neuralrecon (fusion head)
– Custom Ops: ops/back_project, ops/generate_grids, ops/torchsparse_utils
– Checkpointing: saving to checkpoints/
• Inference Pipeline
– Entry: main.py --cfg test.yaml
– Same models + ops → outputs: raw TSDFs → mesh extraction → saved to results/
• Evaluation & Metrics
– tools/evaluation.py (batch eval + pyrender)
– tools/visualize_metrics.py
• Real-time Demo
– demo.py, run_demo_batch.sh
– tools/simple_loader, tools/sync_poses, tools/render → live display loop
• External Libraries & Runtime
– Python, PyTorch, TorchSparse, CUDA, pyrender, conda environment
b. Relationships & interactions
• Raw data → Data Preparation → Processed dataset
• Processed dataset → Training Pipeline → Checkpoints
• Checkpoints + Test data → Inference Pipeline → Results (meshes)
• Results + GT → Evaluation → Metrics
• Live video stream → Demo pipeline → Real-time fused TSDF & render
• Config Manager feeds all pipelines with hyperparameters and file paths.
• Custom Ops are invoked inside both training and inference.
c. Architectural patterns & principles
• Modular separation: data, config, model, ops, tools
• Config-driven workflows (YAML + Python)
• Two-phase training (fragment → fusion) controlled via config flags
• Reuse of common modules for training/inference/demo
• Clear layering: data → model → postprocess
d. Key technologies/frameworks
• PyTorch (torch.distributed), TorchSparse for sparse conv
• Custom C++/CUDA ops wrapped in ops/
• Pyrender/OpenGL for mesh rendering
• Conda for environment reproducibility