How fanegg/feat2gs works
Main Components
• Data Preparation & Dataset Building:
• Key directories: build_dataset (contains scripts for creating JSON splits and building the dataset), scene (handles camera, colmap, and scene-related functions), and assets.
• These modules are responsible for converting raw data into the format required for evaluation and training.
• Model Configuration & Backbone Setup:
• Key directories: configs/backbone lists various YAML config files for different model backbones (like CLIP variants, convnext, deit, dinov2, etc.). This indicates that the system supports multiple feature extraction models.
• Feature Extraction & Initialization:
• Scripts and modules such as DUSt3R initialization and coarse_init_eval.py/…dtu.py indicate the steps for feature extraction.
• The arguments, command, and several Python scripts (e.g., demo.py, demo_run_video.py, demo_train_feat2gs.py) suggest that there is a pipeline for initializing DUSt3R, extracting features, and then transferring them for 3D Gaussian splatting.
• Evaluation & Optimization Pipeline:
• Evaluation scripts (run_feat2gs_eval*.sh in the scripts directory) show a clear multi-step process:
1. DUSt3R initialization & feature extraction.
2. Readout of 3D Gaussian Splatting (3DGS) from features and pose optimization.
3. Pose testing and rendering (using scripts for test view rendering, run_video*, and evals).
• Utilities (in utils, paper, and lpipsPyTorch) support computation of metrics, loss functions, and camera/trajectory handling.
• Rendering & Visualization Component:
• Directories like render.py, render_dep_nom.py, render_dtu.py, and video (with various generate_video scripts) are used to convert 3D scene data into rendered videos or interactive visualizations.
• The gaussian_renderer and its network_gui underline an interactive rendering tool.
• Interactive Demo Component:
• The demo.py and demo_run_video.py scripts as well as the Gradio demo command in the README show a UI front-end that allows users to upload images and view interactive results.
• Submodules & External Dependencies:
• Submodules (e.g., mast3r, dust3r, simple-knn) are external codebases that form essential parts of the pipeline.
• Their directories include additional model implementations, Dockerfiles for deployment, and extra utilities for tasks such as RoPE (Rotary Positional Embedding) acceleration.
Denote Relationships and Data Flow
• From there, show how features flow into the model probing modules, which use the configurations in configs/backbone and code in evals/models.
Architectural Patterns & Design Principles
• The project follows a modular design approach with clear separation between:
• Data ingestion & preparation.
• Model configuration and feature extraction.
• Evaluation/optimization.
• Rendering and interactive display.
• It uses configuration files (in YAML) to enable flexible switching between backbones.
• The GPU lock mechanism in the evaluation scripts suggests a concurrent processing design with attention to resource management.
• The use of submodules indicates adherence to modularity and code reuse principles, keeping third-party components encapsulated.
Technologies, Frameworks, or Libraries
• Python is the main programming language.
• PyTorch for neural network and feature extraction functionalities (installation instructions mention strict versions).
• CUDA kernels in submodules (e.g., submodules/simple-knn and compilation instructions in mast3r/dust3r) indicate GPU acceleration.
• Gradio is used for the interactive demo.
• Additional libraries for image and video processing, metric computation (lpipsPyTorch), and custom utilities under utils.
Putting It All Together
• Continue with a branch that goes to “3D Gaussian Splatting Readout & Pose Optimization” (indicating both DUSt3R initialization and subsequent evaluation steps).
• Finally, attach an “Interactive Demo” component that interfaces with the above pipelines via Gradio.