How mobiuslqm/modgs works
Project Type and Purpose
• This is a research-oriented ML pipeline (“MoDGS”) for reconstructing and rendering dynamic 3D scenes from monocular video.
• Key stages: data preprocessing, two-stage training (3D-aware init + 4D optimization), rendering, and evaluation.
• It embeds/extends four major external toolkits: GeoWizard, Neural-Scene-Flow-Fields (NSFF), GaussianFlow/3D-GS, Omnimotion.
File Structure
• Top-level scripts orchestrate the flow:
• preprocess: ./scripts/preprocess_selfmadeData.sh
• Stage1 train: ./scripts/run_stage1.sh → train_PointTrackGS.py
• Stage2 train: ./scripts/run_stage2.sh → train_PointTrackGS.py (with different flags)
• render: ./scripts/render_vis_cam.sh → render.py / render_PointTrackGS.py
• evaluate: evaluate_metric.py / full_eval.py
• Core modules:
• dataloader: input readers (dataloader/timePcdTable_dataset.py)
• scene: camera & Gaussian scene representations (scene/*.py)
• model: flow and density priors (model/*.py)
• gaussian_renderer: the forward rasterizer (gaussian_renderer/network_gui.py)
• pointTrack: 3D-flow extraction (pointTrack.py, pointTrack_tools.py)
• utils: shared routines (utils/*.py)
• Configs: under configs/PointTrack_IphoneData/...
• Submodules: external libraries live in submodules/GeoWizard, submodules/Neural-Scene-Flow-Fields, submodules/omnimotion.
README for Insights
• Two training stages:
a) 3D-Aware Initialization — uses pointTrack + Gaussian initialization
b) 4D Optimization — rendering + ordinal depth loss
• External dependencies must be installed first (GaussianFlow, 3D-GS, Simple-KNN).
• Scripts glue everything together.
1) Data Ingestion & Preprocessing
• Raw monocular video + camera intrinsics →
• Depth Estimator (Omnimotion / external)
• Exhaustive Flow Estimator (Omnimotion scripts)
• Scale Alignment (preprocess/optimize_scale_shift.py)
• 3D Flow Extraction (pointTrack.py)
2) Training Pipeline
• Stage 1: 3D-Aware Init (train_PointTrackGS.py)
• Modules: model/neuralsceneflowprior.py, Gaussian splatting init
• Dependencies: GeoWizard (for priors), NSFF (for flow priors)
• Stage 2: 4D Optimization (train_PointTrackGS.py + configs)
• Modules: gaussian_renderer, scene/*.py, loss_utils, lpipsPyTorch
3) Checkpoint Storage (file system, model/*.pth)
4) Rendering Pipeline
• render.py / render_PointTrackGS.py invoked by render_vis_cam.sh
• Uses Gaussian renderer + camera samplers (scene/cameras.py)
5) Evaluation
• Metrics scripts: evaluate_metric.py, full_eval.py
• Uses lpipsPyTorch, metrics.py
6) Shared Utilities
• utils/*
• Raw data → Preprocessing → Preprocessed dataset
• Preprocessed dataset → Stage1 Training → Stage1 Checkpoint
• Stage1 Checkpoint + Preprocessed data → Stage2 Training → Final Checkpoint
• Final Checkpoint → Rendering → Rendered Images/Videos
• Rendered Outputs + Ground truth → Evaluation → Metrics
c. Architectural Patterns & Principles
• Pipeline architecture: clear stages chained by scripts
• Modular code: separate folders for data, model, scene, rendering, utils
• Extensibility: external submodules for priors & specialized nets
• Config-driven: everything parameterized via configs/…/*.ini
d. Key Technologies & Frameworks
• Python + PyTorch
• Gaussian splatting (external GaussianFlow, 3D-GS)
• Neural Scene Flow Fields (NSFF)
• GeoWizard for geometry priors
• Omnimotion for motion tracking & flow
• LPIPS for perceptual loss
Elements
• Blue for core modules (model, scene, renderer)
• Green for pipelines (scripts)
• Orange for data stores (datasets, checkpoints)
• Gray clouds for external submodules