How jinlinyi/sparseplanes works
1. This repository is a research codebase for reconstructing a coherent planar 3D scene from two RGB views. The top-level architecture is split into three phases: dataset preparation, model training/inference, and evaluation/visualization. The core implementation is a Detectron2-based PyTorch project inside sparsePlane/.
2. The model itself is organized around a multi-branch detection-and-optimization pipeline. A ResNet50-FPN backbone feeds a plane instance branch, an appearance/embedding branch, and a camera-prediction branch. The repository’s README describes a two-step inference process: first predict planes and relative camera hypotheses, then fuse those discrete predictions and refine them with continuous optimization.
3. The main modeling boundary is in sparsePlane/sparseplane/modeling/. meta_arch/siamese_rcnn.py is the central architecture entry point, with camera_branch.py handling the camera-pose side of the system. roi_heads/ contains the plane-specific heads and losses, including plane_head.py, embedding_head.py, plane_loss.py, and embedding_loss.py, which together produce plane masks, plane parameters, and correspondence features.
4. Camera reasoning is treated as a separate subsystem from plane detection. modeling/camera_net/camera_head.py predicts relative camera transformation probabilities, while utils/camera.py and the precomputed models/kmeans_rots_32.pkl / kmeans_trans_32.pkl support a discrete pose-codebook style representation. This matches the repo’s optimization stage, where the system selects among candidate camera poses before refinement.
5. The data boundary centers on Matterport3D-style inputs. sparseplane/data/datasets/mp3d.py and builtin.py register datasets, while augmentation.py and planercnn_transforms.py adapt image/label transforms for plane detection training. The data_preparation/ folder contains the offline conversion pipeline for MP3D meshes and planes, including scripts to convert .ply, fit planes, and render plane annotations.
6. data_preparation/ is effectively an ETL pipeline for turning raw 3D scene assets into training-ready supervision. Its bash and Python scripts operate on mesh geometry, fit planar segments, and produce rendered outputs used by the learning pipeline. This is a major architectural boundary because the training data is not consumed directly from raw scans; it is preprocessed into plane-centric labels first.
7. Training and inference are exposed as command-line tools under sparsePlane/tools/. train_net.py is the training entry point, inference_sparse_plane.py runs reconstruction on image pairs, eval.py performs evaluation, and visualize_gt.py generates ground-truth visualizations. These scripts sit above the Detectron2-style config system and model registry, which is why sparseplane/config/config.py and the YAML configs are important orchestration points.
8. The configuration layer is explicit and split by stage. sparsePlane/configs/step1-plane.yaml, step2-corr.yaml, and step3-camera.yaml suggest a staged workflow: plane estimation, correspondence/association, then camera estimation. That staging is important architecturally because it mirrors the system’s decomposition into separate predictors and downstream fusion.
9. Evaluation is separated from training and uses dataset-specific metrics. sparseplane/evaluation/mp3d_evaluation.py and detectron2coco.py indicate COCO-style evaluation adapters plus Matterport3D-specific scoring. Supporting metrics and geometry utilities live under sparseplane/utils/metrics.py, mesh_utils.py, and meshes.py, showing that evaluation includes both segmentation-style and 3D reconstruction-style outputs.
10. Visualization and geometry utilities form the last major subsystem. sparseplane/visualization/visualization.py, utils/vis.py, textures.py, and mesh helpers generate rendered plane overlays and reconstructed scene views. These are not core learning components, but they are essential for interpreting the pipeline’s output: detected planes, correspondences, selected camera pose, and the final reconstructed planar surface.
11. The repo depends on Detectron2 conventions for model registration, data loaders, and training loops, with Python packaging configured under sparsePlane/setup.py and setup.cfg. The root requirements.txt plus the environment docs indicate a PyTorch-centric runtime, while bash wrappers in data_preparation/ and tools/ show that most workflows are executed as CLI jobs rather than through a long-running service.
12. In architecture-diagram terms, the cleanest flow is: raw scan/scene assets enter data_preparation, become plane-labeled datasets, flow into Detectron2-based training and inference in sparseplane/modeling, then pass through camera/plane fusion and refinement, and finally out through evaluation and visualization utilities. The key boundary to preserve is that plane detection, camera estimation, and geometric optimization are separate subsystems coordinated by the inference scripts rather than one monolithic network.
Main entry points: siamese_rcnn.py Siamese RCNN · inference_sparse_plane.py Infer · roi_heads.py ROI heads · data_preparation Data prep
Components 19
Data Prep
Offline ETL from raw Matterport3D assets to plane-labeled training data
- Data prep · Converts raw scene assets into plane-centric supervision
data_preparation - Convert PLY · Turns MP3D meshes into intermediate training assets
data_preparation/convert_mp3d_ply.py - Fit planes · Derives planar segments and annotations from geometry
data_preparation/fit_plane_ply.py - Render planes · Produces rendered plane supervision and previews
data_preparation/render_plane.py
Core Model
Detectron2-based plane, correspondence, and camera prediction stack
- Config · Detectron2-style config and staged YAML workflow
sparsePlane/sparseplane/config/config.py - Siamese RCNN · Main two-view reconstruction architecture
sparsePlane/sparseplane/modeling/meta_arch/siamese_rcnn.py - Camera branch · Coordinates the camera-side path inside the meta-architecture
sparsePlane/sparseplane/modeling/meta_arch/camera_branch.py - ROI heads · Plane and embedding heads plus associated losses
sparsePlane/sparseplane/modeling/roi_heads/roi_heads.py - Plane head · Predicts plane masks and plane parameters
sparsePlane/sparseplane/modeling/roi_heads/plane_head.py - Embedding head · Builds correspondence features for plane association
sparsePlane/sparseplane/modeling/roi_heads/embedding_head.py - Camera head · Predicts discrete relative camera transformation probabilities
sparsePlane/sparseplane/modeling/camera_net/camera_head.py - Depth head · Auxiliary depth prediction branch
sparsePlane/sparseplane/modeling/depth_net/depth_head.py - Camera utils · Discrete pose utilities backed by camera codebooks
sparsePlane/sparseplane/utils/camera.py
Tools
Training, inference, and demo entry points
Eval & Viz
Metrics, reconstruction utilities, and rendering/inspection
- Eval tool · Runs dataset-aware evaluation workflows
sparsePlane/tools/eval.py - MP3D eval · Matterport3D-specific evaluation logic
sparsePlane/sparseplane/evaluation/mp3d_evaluation.py - Visualization · Renders plane overlays, reconstructions, and inspection outputs
sparsePlane/sparseplane/visualization/visualization.py - Mesh utils · Mesh and reconstruction helpers used by evaluation and visualization
sparsePlane/sparseplane/utils/mesh_utils.py
Connections 22
- Data prep → Convert PLY: preprocesses · Raw scan assets are converted into usable mesh inputs
- Data prep → Fit planes: fits · Geometry is segmented into planar supervision
- Data prep → Render planes: renders · Plane annotations and previews are generated offline
- Fit planes → Render planes: feeds · Fitted planes are rendered into training artifacts
- Config → Siamese RCNN: configures · Stage configs and registry drive model assembly
- Siamese RCNN → ROI heads: uses · Central architecture delegates plane prediction to ROI heads
- Siamese RCNN → Camera branch: routes · Camera reasoning is handled as a separate branch
- Siamese RCNN → Camera head: predicts · Relative pose hypotheses are produced by the camera net
- Siamese RCNN → Depth head: auxiliary · Optional depth supervision supports reconstruction
- ROI heads → Plane head: predicts · Plane masks and plane parameters are produced here
- ROI heads → Embedding head: associates · Correspondence embeddings support plane matching
- ROI heads → Camera utils: codes · Discrete pose codebooks support camera selection
- Train → Config: loads · Training is driven by stage-aware config files
- Train → Siamese RCNN: trains · Training entry point executes the main model
- Infer → Siamese RCNN: runs · Inference performs two-view plane reconstruction
- Infer → Camera utils: selects · Candidate poses are drawn from discrete camera hypotheses
- Infer → Mesh utils: refines · Continuous geometric refinement uses mesh helpers
- Eval tool → MP3D eval: delegates · Evaluation tool dispatches to dataset-specific scoring
- MP3D eval → Mesh utils: measures · 3D geometry utilities support reconstruction metrics
- Visualization → Mesh utils: renders · Mesh helpers underpin scene reconstruction views
- Eval tool → Visualization: visualizes · Evaluation outputs can be inspected graphically
- Infer → Visualization: exports · Inference results are visualized as plane and camera outputs