# fanegg/feat2gs architecture

> Architecture diagram of the GitHub repository fanegg/feat2gs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fanegg/feat2gs
- Repository: https://github.com/fanegg/feat2gs

## Overview

##### Main Components
- Data Preparation & Dataset Building:
- Key directories: build_dataset (contains scripts for creating JSON splits and building the dataset), scene (handles camera, colmap, and scene-related functions), and assets.
- These modules are responsible for converting raw data into the format required for evaluation and training.

- Model Configuration & Backbone Setup:
- Key directories: configs/backbone lists various YAML config files for different model backbones (like CLIP variants, convnext, deit, dinov2, etc.). This indicates that the system supports multiple feature extraction models.

- Feature Extraction & Initialization:
- Scripts and modules such as DUSt3R initialization and coarse_init_eval.py/…dtu.py indicate the steps for feature extraction.
- The arguments, command, and several Python scripts (e.g., demo.py, demo_run_video.py, demo_train_feat2gs.py) suggest that there is a pipeline for initializing DUSt3R, extracting features, and then transferring them for 3D Gaussian splatting.

- Evaluation & Optimization Pipeline:
- Evaluation scripts (run_feat2gs_eval*.sh in the scripts directory) show a clear multi-step process:
1. DUSt3R initialization & feature extraction.
2. Readout of 3D Gaussian Splatting (3DGS) from features and pose optimization.
3. Pose testing and rendering (using scripts for test view rendering, run_video*, and evals).
- Utilities (in utils, paper, and lpipsPyTorch) support computation of metrics, loss functions, and camera/trajectory handling.

- Rendering & Visualization Component:
- Directories like render.py, render_dep_nom.py, render_dtu.py, and video (with various generate_video scripts) are used to convert 3D scene data into rendered videos or interactive visualizations.
- The gaussian_renderer and its network_gui underline an interactive rendering tool.

- Interactive Demo Component:
- The demo.py and demo_run_video.py scripts as well as the Gradio demo command in the README show a UI front-end that allows users to upload images and view interactive results.

- Submodules & External Dependencies:
- Submodules (e.g., mast3r, dust3r, simple-knn) are external codebases that form essential parts of the pipeline.
- Their directories include additional model implementations, Dockerfiles for deployment, and extra utilities for tasks such as RoPE (Rotary Positional Embedding) acceleration.

##### Denote Relationships and Data Flow
- From there, show how features flow into the model probing modules, which use the configurations in configs/backbone and code in evals/models.

##### Architectural Patterns & Design Principles
- The project follows a modular design approach with clear separation between:
- Data ingestion & preparation.
- Model configuration and feature extraction.
- Evaluation/optimization.
- Rendering and interactive display.
- It uses configuration files (in YAML) to enable flexible switching between backbones.
- The GPU lock mechanism in the evaluation scripts suggests a concurrent processing design with attention to resource management.
- The use of submodules indicates adherence to modularity and code reuse principles, keeping third-party components encapsulated.

##### Technologies, Frameworks, or Libraries
- Python is the main programming language.
- PyTorch for neural network and feature extraction functionalities (installation instructions mention strict versions).
- CUDA kernels in submodules (e.g., submodules/simple-knn and compilation instructions in mast3r/dust3r) indicate GPU acceleration.
- Gradio is used for the interactive demo.
- Additional libraries for image and video processing, metric computation (lpipsPyTorch), and custom utilities under utils.

##### Putting It All Together
- Continue with a branch that goes to “3D Gaussian Splatting Readout & Pose Optimization” (indicating both DUSt3R initialization and subsequent evaluation steps).
- Finally, attach an “Interactive Demo” component that interfaces with the above pipelines via Gradio.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Start
    DI["Data Input"]:::data

    %% Data Preparation & Ingestion Subgraph
    subgraph "Data Preparation & Ingestion"
        BD["build_dataset"]:::data
        SC["scene"]:::data
        AS["assets"]:::data
    end

    %% Feature Extraction & Initialization Subgraph
    subgraph "Feature Extraction & Initialization"
        CE["coarse_init_eval.py"]:::extraction
        DT["demo_train_feat2gs.py"]:::extraction
    end

    %% Model Configuration & Backbone Setup Subgraph
    subgraph "Model Configuration"
        CB["configs/backbone"]:::config
        EM["evals/models"]:::config
    end

    %% Evaluation & Optimization Pipeline Subgraph
    subgraph "Evaluation & Optimization"
        RE["run_feat2gs_eval.sh"]:::evaluation
        RP["run_feat2gs_eval_parallel.sh"]:::evaluation
        RD["run_feat2gs_eval_dtu_parallel.sh"]:::evaluation
    end

    %% Rendering & Visualization Subgraph
    subgraph "Rendering & Visualization"
        R1["render.py"]:::rendering
        R2["render_dep_nom.py"]:::rendering
        R3["render_dtu.py"]:::rendering
        GV["generate_video.py"]:::rendering
        NG["network_gui.py"]:::rendering
    end

    %% Interactive Demo Component Subgraph
    subgraph "Interactive Demo"
        D1["demo.py"]:::demo
        D2["demo_run_video.py"]:::demo
    end

    %% External Submodules Subgraph
    subgraph "External Submodules"
        M["mast3r"]:::external
        SK["simple-knn"]:::external
    end

    %% Connections
    DI -->|"feeds"| BD
    DI --> SC
    DI --> AS

    BD -->|"prepares data"| CE
    SC --> CE
    AS --> CE

    CE -->|"extracts features"| DT
    DT -->|"passes features"| CB
    CB -->|"configures models"| EM
    EM -->|"probes model"| RE

    RE --> RP
    RP --> RD
    RD -->|"optimizes pose"| R1
    R1 --> R2
    R2 --> R3
    R3 --> GV
    GV --> NG
    NG -->|"renders output"| D1
    NG --> D2

    %% External integrations
    M ---|"integration"| CE
    SK ---|"integration"| RE

    %% Class Definitions for colouring
    classDef data fill:#A7C7E7,stroke:#000,stroke-width:2px;
    classDef extraction fill:#B2E4B2,stroke:#000,stroke-width:2px;
    classDef config fill:#C2EABD,stroke:#000,stroke-width:2px;
    classDef evaluation fill:#FEEFB3,stroke:#000,stroke-width:2px;
    classDef rendering fill:#FFD2A1,stroke:#000,stroke-width:2px;
    classDef demo fill:#D1C0E8,stroke:#000,stroke-width:2px;
    classDef external fill:#E0E0E0,stroke:#000,stroke-width:2px;

    %% Click Events
    click BD "https://github.com/fanegg/feat2gs/tree/main/build_dataset"
    click SC "https://github.com/fanegg/feat2gs/tree/main/scene"
    click AS "https://github.com/fanegg/feat2gs/tree/main/assets"
    click CB "https://github.com/fanegg/feat2gs/tree/main/configs/backbone"
    click EM "https://github.com/fanegg/feat2gs/tree/main/evals/models"
    click CE "https://github.com/fanegg/feat2gs/blob/main/coarse_init_eval.py"
    click DT "https://github.com/fanegg/feat2gs/blob/main/demo_train_feat2gs.py"
    click D1 "https://github.com/fanegg/feat2gs/blob/main/demo.py"
    click D2 "https://github.com/fanegg/feat2gs/blob/main/demo_run_video.py"
    click R1 "https://github.com/fanegg/feat2gs/blob/main/render.py"
    click R2 "https://github.com/fanegg/feat2gs/blob/main/render_dep_nom.py"
    click R3 "https://github.com/fanegg/feat2gs/blob/main/render_dtu.py"
    click GV "https://github.com/fanegg/feat2gs/blob/main/video/generate_video.py"
    click NG "https://github.com/fanegg/feat2gs/blob/main/gaussian_renderer/network_gui.py"
    click RE "https://github.com/fanegg/feat2gs/blob/main/scripts/run_feat2gs_eval.sh"
    click RP "https://github.com/fanegg/feat2gs/blob/main/scripts/run_feat2gs_eval_parallel.sh"
    click RD "https://github.com/fanegg/feat2gs/blob/main/scripts/run_feat2gs_eval_dtu_parallel.sh"
    click M "https://github.com/fanegg/feat2gs/tree/main/submodules/mast3r"
    click SK "https://github.com/fanegg/feat2gs/tree/main/submodules/simple-knn"
```
