# zju3dv/neuralrecon architecture

> Architecture diagram of the GitHub repository zju3dv/neuralrecon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zju3dv/neuralrecon
- Repository: https://github.com/zju3dv/neuralrecon

## Overview

##### Project type and purpose
- NeuralRecon is a research‐code full-stack machine-learning pipeline for real-time coherent 3D reconstruction from monocular video.
- It includes data preparation, model training, inference, evaluation, and a live demo (including ARKit support).

##### File structure
- Top-level scripts (main.py, demo.py, train.sh, test.sh, run_demo_batch.sh) drive each stage.
- config/: YAML/Python configs for train/val/test/demo.
- datasets/: data-loading, sampling, transforms for ScanNet and ARKit.
- models/: network definitions (backbone, GRU fusion, NeuCon network, full recon head).
- ops/: CUDA/PyTorch custom ops (back-projection, TSDF grid gen, TorchSparse helpers).
- tools/: preprocessing (Generate GT TSDFs), postprocessing (evaluation, render, metrics, ARKit utilities).
- assets/: demo GIFs, visual examples.
- environment.yaml, requirements.txt: dependency management.
- checkpoints/, results/: where pretrained weights and outputs live.

##### README insights
- Describes training phases (fragment‐level then GRU fusion).
- Inference pipeline produces meshes then evaluation via pyrender.
- Real-time demo uses camera streams + live fusion.
- Data prep scripts generate ground-truth TSDF volumes using tools/tsdf_fusion.

a. Main components
- Data Sources
– ScanNet raw images + poses
– Custom ARKit video + poses
- Data Preparation
– tools/tsdf_fusion/generate_gt.py → ground-truth TSDFs
– datasets/…: samplers, transforms → PyTorch DataLoader
- Configuration Manager
– config/*.yaml/python → global hyperparameters & paths
- Training Pipeline
– Entry: train.sh → torch.distributed launch → main.py (train mode)
– Models: models/backbone → models/neucon_network & models/gru_fusion → models/neuralrecon (fusion head)
– Custom Ops: ops/back_project, ops/generate_grids, ops/torchsparse_utils
– Checkpointing: saving to checkpoints/
- Inference Pipeline
– Entry: main.py --cfg test.yaml
– Same models + ops → outputs: raw TSDFs → mesh extraction → saved to results/
- Evaluation & Metrics
– tools/evaluation.py (batch eval + pyrender)
– tools/visualize_metrics.py
- Real-time Demo
– demo.py, run_demo_batch.sh
– tools/simple_loader, tools/sync_poses, tools/render → live display loop
- External Libraries & Runtime
– Python, PyTorch, TorchSparse, CUDA, pyrender, conda environment

b. Relationships & interactions
- Raw data → Data Preparation → Processed dataset
- Processed dataset → Training Pipeline → Checkpoints
- Checkpoints + Test data → Inference Pipeline → Results (meshes)
- Results + GT → Evaluation → Metrics
- Live video stream → Demo pipeline → Real-time fused TSDF & render
- Config Manager feeds all pipelines with hyperparameters and file paths.
- Custom Ops are invoked inside both training and inference.

c. Architectural patterns & principles
- Modular separation: data, config, model, ops, tools
- Config-driven workflows (YAML + Python)
- Two-phase training (fragment → fusion) controlled via config flags
- Reuse of common modules for training/inference/demo
- Clear layering: data → model → postprocess

d. Key technologies/frameworks
- PyTorch (torch.distributed), TorchSparse for sparse conv
- Custom C++/CUDA ops wrapped in ops/
- Pyrender/OpenGL for mesh rendering
- Conda for environment reproducibility

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    subgraph "External Systems"
        direction TB
        ScanNet["ScanNet Dataset"]:::external
        ARKit["ARKit Dataset"]:::external
        CondaEnv["Conda Environment (environment.yaml)"]:::external
        PyReq["Python Dependencies (requirements.txt)"]:::external
    end

    %% Data Sources
    subgraph "Data Sources" 
        direction TB
        SNLoader["ScanNet Loader"]:::core
        ARLoader["ARKit Loader"]:::core
    end

    %% Data Preparation
    subgraph "Data Preparation" 
        direction TB
        GTGen["GT TSDF Generation"]:::tools
        FusionUtils["Fusion Utilities"]:::tools
        Sampler["Data Sampler"]:::core
        Transforms["Data Transforms"]:::core
    end

    %% Configuration Manager
    subgraph "Configuration Manager"
        direction TB
        DefaultCfg["Global Defaults (default.py)"]:::config
        TrainCfg["Training Config (train.yaml)"]:::config
        TestCfg["Testing Config (test.yaml)"]:::config
        DemoCfg["Demo Config (demo.yaml)"]:::config
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        TrainScript["train.sh"]:::script
        MainTrain["main.py (train mode)"]:::script
        Backbone["Backbone Model"]:::core
        NeuConNet["NeuCon Network"]:::core
        GRUFusion["GRU Fusion Module"]:::core
        ReconHead["Reconstruction Head"]:::core
        BackProj["Back-Projection Op"]:::core
        GenGrids["Grid Generation Op"]:::core
        TSparse["TorchSparse Utils"]:::core
        Checkpoints["checkpoints/"]:::storage
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        TestScript["test.sh"]:::script
        MainInfer["main.py (--cfg test.yaml)"]:::script
        ResultsDir["results/"]:::storage
    end

    %% Evaluation & Metrics
    subgraph "Evaluation & Metrics"
        direction TB
        EvalBatch["tools/evaluation.py"]:::tools
        EvalUtils["tools/evaluation_utils.py"]:::tools
        VizMetrics["tools/visualize_metrics.py"]:::tools
    end

    %% Real-time Demo
    subgraph "Demo Loop"
        direction TB
        DemoScript["demo.py"]:::script
        RunDemo["run_demo_batch.sh"]:::script
        SimpleLoader["tools/simple_loader.py"]:::tools
        SyncPoses["tools/sync_poses.py"]:::tools
        ProcessAR["tools/process_arkit_data.py"]:::tools
        RenderTool["tools/render.py"]:::tools
    end

    %% External Utils
    subgraph "Utilities"
        direction TB
        Utils["utils.py"]:::core
    end

    %% Data Flow
    ScanNet --> SNLoader
    ARKit --> ARLoader
    SNLoader --> Sampler
    ARLoader --> Sampler
    Sampler --> Transforms
    Transforms --> GTGen
    Transforms --> FusionUtils
    GTGen --> TrainScript
    FusionUtils --> TrainScript

    %% Config to Pipelines
    DefaultCfg --> TrainScript
    TrainCfg --> TrainScript
    DefaultCfg --> TestScript
    TestCfg --> TestScript
    DefaultCfg --> DemoScript
    DemoCfg --> DemoScript

    %% Training connections
    TrainScript --> MainTrain
    MainTrain --> Backbone
    MainTrain --> NeuConNet
    MainTrain --> GRUFusion
    MainTrain --> ReconHead
    Backbone --> BackProj
    NeuConNet --> GenGrids
    GRUFusion --> TSparse
    MainTrain --> Checkpoints

    %% Inference connections
    TestScript --> MainInfer
    MainInfer --> Backbone
    MainInfer --> NeuConNet
    MainInfer --> GRUFusion
    MainInfer --> ReconHead
    MainInfer --> Checkpoints
    MainInfer --> ResultsDir

    %% Evaluation flow
    ResultsDir --> EvalBatch
    EvalBatch --> EvalUtils
    EvalBatch --> VizMetrics

    %% Demo loop flow
    DemoScript --> SimpleLoader
    SimpleLoader --> SyncPoses
    SyncPoses --> ProcessAR
    ProcessAR --> RenderTool
    RenderTool --> DemoScript

    %% External deps
    CondaEnv --> MainTrain
    PyReq --> MainTrain
    CondaEnv --> MainInfer
    PyReq --> MainInfer
    CondaEnv --> DemoScript
    PyReq --> DemoScript

    %% Click Events
    click SNLoader "https://github.com/zju3dv/neuralrecon/blob/master/datasets/scannet.py"
    click ARLoader "https://github.com/zju3dv/neuralrecon/blob/master/datasets/demo.py"
    click GTGen "https://github.com/zju3dv/neuralrecon/blob/master/tools/tsdf_fusion/generate_gt.py"
    click FusionUtils "https://github.com/zju3dv/neuralrecon/blob/master/tools/tsdf_fusion/fusion.py"
    click Sampler "https://github.com/zju3dv/neuralrecon/blob/master/datasets/sampler.py"
    click Transforms "https://github.com/zju3dv/neuralrecon/blob/master/datasets/transforms.py"
    click DefaultCfg "https://github.com/zju3dv/neuralrecon/blob/master/config/default.py"
    click TrainCfg "https://github.com/zju3dv/neuralrecon/blob/master/config/train.yaml"
    click TestCfg "https://github.com/zju3dv/neuralrecon/blob/master/config/test.yaml"
    click DemoCfg "https://github.com/zju3dv/neuralrecon/blob/master/config/demo.yaml"
    click TrainScript "https://github.com/zju3dv/neuralrecon/blob/master/train.sh"
    click MainTrain "https://github.com/zju3dv/neuralrecon/blob/master/main.py"
    click Backbone "https://github.com/zju3dv/neuralrecon/blob/master/models/backbone.py"
    click NeuConNet "https://github.com/zju3dv/neuralrecon/blob/master/models/neucon_network.py"
    click GRUFusion "https://github.com/zju3dv/neuralrecon/blob/master/models/gru_fusion.py"
    click ReconHead "https://github.com/zju3dv/neuralrecon/blob/master/models/neuralrecon.py"
    click BackProj "https://github.com/zju3dv/neuralrecon/blob/master/ops/back_project.py"
    click GenGrids "https://github.com/zju3dv/neuralrecon/blob/master/ops/generate_grids.py"
    click TSparse "https://github.com/zju3dv/neuralrecon/blob/master/ops/torchsparse_utils.py"
    click TestScript "https://github.com/zju3dv/neuralrecon/blob/master/test.sh"
    click EvalBatch "https://github.com/zju3dv/neuralrecon/blob/master/tools/evaluation.py"
    click EvalUtils "https://github.com/zju3dv/neuralrecon/blob/master/tools/evaluation_utils.py"
    click VizMetrics "https://github.com/zju3dv/neuralrecon/blob/master/tools/visualize_metrics.py"
    click DemoScript "https://github.com/zju3dv/neuralrecon/blob/master/demo.py"
    click RunDemo "https://github.com/zju3dv/neuralrecon/blob/master/run_demo_batch.sh"
    click SimpleLoader "https://github.com/zju3dv/neuralrecon/blob/master/tools/simple_loader.py"
    click SyncPoses "https://github.com/zju3dv/neuralrecon/blob/master/tools/sync_poses.py"
    click ProcessAR "https://github.com/zju3dv/neuralrecon/blob/master/tools/process_arkit_data.py"
    click RenderTool "https://github.com/zju3dv/neuralrecon/blob/master/tools/render.py"
    click Utils "https://github.com/zju3dv/neuralrecon/blob/master/utils.py"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef tools fill:#9ACD32,stroke:#000,stroke-width:1px
    classDef script fill:#FFA500,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef config fill:#FFE4B5,stroke:#000,stroke-width:1px
    classDef storage fill:#EEE8AA,stroke:#000,stroke-width:1px
```
