# mobiuslqm/modgs architecture

> Architecture diagram of the GitHub repository mobiuslqm/modgs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mobiuslqm/modgs
- Repository: https://github.com/mobiuslqm/modgs

## Overview

##### Project Type and Purpose
- This is a research-oriented ML pipeline (“MoDGS”) for reconstructing and rendering dynamic 3D scenes from monocular video.
- Key stages: data preprocessing, two-stage training (3D-aware init + 4D optimization), rendering, and evaluation.
- It embeds/extends four major external toolkits: GeoWizard, Neural-Scene-Flow-Fields (NSFF), GaussianFlow/3D-GS, Omnimotion.

##### File Structure
- Top-level scripts orchestrate the flow:
- preprocess: `./scripts/preprocess_selfmadeData.sh`
- Stage1 train: `./scripts/run_stage1.sh` → `train_PointTrackGS.py`
- Stage2 train: `./scripts/run_stage2.sh` → `train_PointTrackGS.py` (with different flags)
- render: `./scripts/render_vis_cam.sh` → `render.py` / `render_PointTrackGS.py`
- evaluate: `evaluate_metric.py` / `full_eval.py`
- Core modules:
- dataloader: input readers (`dataloader/timePcdTable_dataset.py`)
- scene: camera & Gaussian scene representations (`scene/*.py`)
- model: flow and density priors (`model/*.py`)
- gaussian_renderer: the forward rasterizer (`gaussian_renderer/network_gui.py`)
- pointTrack: 3D-flow extraction (`pointTrack.py`, `pointTrack_tools.py`)
- utils: shared routines (`utils/*.py`)
- Configs: under `configs/PointTrack_IphoneData/...`
- Submodules: external libraries live in `submodules/GeoWizard`, `submodules/Neural-Scene-Flow-Fields`, `submodules/omnimotion`.

##### README for Insights
- Two training stages:
a) 3D-Aware Initialization — uses pointTrack + Gaussian initialization
b) 4D Optimization — rendering + ordinal depth loss
- External dependencies must be installed first (GaussianFlow, 3D-GS, Simple-KNN).
- Scripts glue everything together.

1) Data Ingestion & Preprocessing
- Raw monocular video + camera intrinsics →
- Depth Estimator (Omnimotion / external)
- Exhaustive Flow Estimator (Omnimotion scripts)
- Scale Alignment (`preprocess/optimize_scale_shift.py`)
- 3D Flow Extraction (`pointTrack.py`)
2) Training Pipeline
- Stage 1: 3D-Aware Init (`train_PointTrackGS.py`)
- Modules: model/neuralsceneflowprior.py, Gaussian splatting init
- Dependencies: GeoWizard (for priors), NSFF (for flow priors)
- Stage 2: 4D Optimization (`train_PointTrackGS.py` + configs)
- Modules: gaussian_renderer, scene/*.py, loss_utils, lpipsPyTorch
3) Checkpoint Storage (file system, model/*.pth)
4) Rendering Pipeline
- `render.py` / `render_PointTrackGS.py` invoked by `render_vis_cam.sh`
- Uses Gaussian renderer + camera samplers (`scene/cameras.py`)
5) Evaluation
- Metrics scripts: `evaluate_metric.py`, `full_eval.py`
- Uses lpipsPyTorch, metrics.py
6) Shared Utilities
- utils/*

- Raw data → Preprocessing → Preprocessed dataset
- Preprocessed dataset → Stage1 Training → Stage1 Checkpoint
- Stage1 Checkpoint + Preprocessed data → Stage2 Training → Final Checkpoint
- Final Checkpoint → Rendering → Rendered Images/Videos
- Rendered Outputs + Ground truth → Evaluation → Metrics

c. Architectural Patterns & Principles
- Pipeline architecture: clear stages chained by scripts
- Modular code: separate folders for data, model, scene, rendering, utils
- Extensibility: external submodules for priors & specialized nets
- Config-driven: everything parameterized via `configs/…/*.ini`

d. Key Technologies & Frameworks
- Python + PyTorch
- Gaussian splatting (external GaussianFlow, 3D-GS)
- Neural Scene Flow Fields (NSFF)
- GeoWizard for geometry priors
- Omnimotion for motion tracking & flow
- LPIPS for perceptual loss

##### Elements
- Blue for core modules (model, scene, renderer)
- Green for pipelines (scripts)
- Orange for data stores (datasets, checkpoints)
- Gray clouds for external submodules

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        PreprocessScript["Preprocess Shell Script"]:::pipeline
        ScaleAlign["Scale Alignment Module"]:::core
        FlowExtract["3D Flow Extraction Module"]:::core
        PreprocessScript -->|".mp4 video, intrinsics"| ScaleAlign
        ScaleAlign -->|"scaled data"| FlowExtract
        FlowExtract -->|"point clouds (.npz)"| PreprocessedData
    end

    %% Data Store
    PreprocessedData[(Preprocessed Dataset)]:::datastore

    %% Training Pipeline
    subgraph "Stage 1 Training"
        direction TB
        Stage1Script["Stage1 Training Script"]:::pipeline
        Stage1Entry["train_PointTrackGS.py"]:::core
        Stage1Script -->|runs| Stage1Entry
        Stage1Entry -->|"initial checkpoint (.pth)"| Stage1Checkpoint
    end

    Stage1Checkpoint[(Stage1 Checkpoint)]:::datastore
    PreprocessedData -->|".npz point clouds"| Stage1Script

    subgraph "Stage 2 Training"
        direction TB
        Stage2Script["Stage2 Training Script"]:::pipeline
        Stage2Entry["train_PointTrackGS.py"]:::core
        Stage2Script -->|runs| Stage2Entry
        Stage2Entry -->|"final checkpoint (.pth)"| FinalCheckpoint
    end

    Stage2Script -->|uses Stage1 checkpoint| Stage2Entry
    PreprocessedData -->|".npz point clouds"| Stage2Script
    Stage1Checkpoint --> Stage2Script

    FinalCheckpoint[(Final Checkpoint)]:::datastore

    %% Rendering Pipeline
    subgraph "Rendering"
        direction TB
        RenderScript["Rendering Shell Script"]:::pipeline
        RenderMain["render_PointTrackGS.py"]:::core
        RenderAlt["render.py"]:::core
        RenderVid["render_video.py"]:::core
        RenderScript -->|invokes| RenderMain
        RenderScript -->|invokes| RenderAlt
        RenderMain -->|outputs frames| RenderedOutput
        RenderAlt -->|outputs frames| RenderedOutput
        RenderVid -->|encodes video| RenderedOutput
    end

    RenderedOutput[(Rendered Images / Videos)]:::datastore
    FinalCheckpoint -->|".pth checkpoint"| RenderScript

    %% Evaluation
    subgraph "Evaluation"
        direction TB
        EvalMetric["evaluate_metric.py"]:::pipeline
        FullEval["full_eval.py"]:::pipeline
        RenderedOutput -->|images, videos| EvalMetric
        EvalMetric -->|scores| Metrics
        RenderedOutput -->|images, videos| FullEval
        FullEval -->|detailed metrics| Metrics
    end

    Metrics[(Evaluation Metrics)]:::datastore

    %% External Toolkits
    subgraph "External Submodules"
        direction TB
        GeoWizard["GeoWizard"]:::external
        NSFF["Neural-Scene-Flow-Fields"]:::external
        Omnimotion["Omnimotion"]:::external
    end

    %% Shared Utilities
    subgraph "Shared Utilities"
        direction TB
        Utils1["utils/loss_utils.py"]:::core
        Utils2["utils/general_utils.py"]:::core
        Utils3["utils/render_utils.py"]:::core
        Utils4["utils/camera_utils.py"]:::core
        Utils5["utils/flow_viz.py"]:::core
        Utils6["utils/gaussian_training_utils.py"]:::core
        Utils7["utils/system_utils.py"]:::core
        Utils8["utils/colormap.py"]:::core
    end

    %% Connections to externals and utils
    Stage1Entry -->|"calls priors"| GeoWizard
    Stage1Entry -->|"calls flow priors"| NSFF
    Stage2Entry -->|"uses gaussian renderer"| GaussianRenderer
    RenderMain -->|"uses camera models"| Cameras
    RenderAlt -->|"uses camera models"| Cameras

    %% Scene & Renderer Modules
    subgraph "Scene & Renderer"
        direction TB
        Cameras["scene/cameras.py"]:::core
        GaussianModel["scene/gaussian_model.py"]:::core
        OriginalGModel["scene/original_gaussian_model.py"]:::core
        IsoGModel["scene/iso_gaussian_model.py"]:::core
        PTGModel["scene/PointTrack_gaussian_model.py"]:::core
        ColmapLoader["scene/colmap_loader.py"]:::core
        DatasetReaders["scene/dataset_readers.py"]:::core
        HyperCam["scene/hyper_camera.py"]:::core
        IPHCam["scene/HexPlaneCodes/iphone_data_loader/camera.py"]:::core
        GaussianRenderer["gaussian_renderer/network_gui.py"]:::core
    end

    %% Data flow to scene
    Stage2Entry -->|"samples camera"| Cameras
    Stage2Entry -->|"renders Gaussian splats"| GaussianRenderer

    %% Click Events
    click PreprocessScript "https://github.com/mobiuslqm/modgs/blob/main/scripts/preprocess_selfmadeData.sh"
    click ScaleAlign "https://github.com/mobiuslqm/modgs/blob/main/preprocess/optimize_scale_shift.py"
    click FlowExtract "https://github.com/mobiuslqm/modgs/blob/main/pointTrack.py"
    click Stage1Script "https://github.com/mobiuslqm/modgs/blob/main/scripts/run_stage1.sh"
    click Stage1Entry "https://github.com/mobiuslqm/modgs/blob/main/train_PointTrackGS.py"
    click Stage2Script "https://github.com/mobiuslqm/modgs/blob/main/scripts/run_stage1.sh"
    click Stage2Entry "https://github.com/mobiuslqm/modgs/blob/main/train_PointTrackGS.py"
    click RenderScript "https://github.com/mobiuslqm/modgs/blob/main/scripts/render_vis_cam.sh"
    click RenderMain "https://github.com/mobiuslqm/modgs/blob/main/render_PointTrackGS.py"
    click RenderAlt "https://github.com/mobiuslqm/modgs/blob/main/render.py"
    click RenderVid "https://github.com/mobiuslqm/modgs/blob/main/render_video.py"
    click EvalMetric "https://github.com/mobiuslqm/modgs/blob/main/evaluate_metric.py"
    click FullEval "https://github.com/mobiuslqm/modgs/blob/main/full_eval.py"
    click Cameras "https://github.com/mobiuslqm/modgs/blob/main/scene/cameras.py"
    click GaussianModel "https://github.com/mobiuslqm/modgs/blob/main/scene/gaussian_model.py"
    click OriginalGModel "https://github.com/mobiuslqm/modgs/blob/main/scene/original_gaussian_model.py"
    click IsoGModel "https://github.com/mobiuslqm/modgs/blob/main/scene/iso_gaussian_model.py"
    click PTGModel "https://github.com/mobiuslqm/modgs/blob/main/scene/PointTrack_gaussian_model.py"
    click ColmapLoader "https://github.com/mobiuslqm/modgs/blob/main/scene/colmap_loader.py"
    click DatasetReaders "https://github.com/mobiuslqm/modgs/blob/main/scene/dataset_readers.py"
    click HyperCam "https://github.com/mobiuslqm/modgs/blob/main/scene/hyper_camera.py"
    click IPHCam "https://github.com/mobiuslqm/modgs/blob/main/scene/HexPlaneCodes/iphone_data_loader/camera.py"
    click GaussianRenderer "https://github.com/mobiuslqm/modgs/blob/main/gaussian_renderer/network_gui.py"

    %% Styles
    classDef core fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef pipeline fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef datastore fill:#ffe0b2,stroke:#ef6c00,color:#e65100
    classDef external fill:#e0e0e0,stroke:#9e9e9e,color:#424242
```
