How half-potato/ever_training works
Project Type and Purpose
• This project implements a real-time differentiable emission-only volume rendering approach based on “Exact Volumetric Ellipsoid Rendering” (EVER). Its purpose is to train, render, and evaluate 3D scenes using a primitive-based, exactly computed volume rendering method, offering advantages over methods like 3D Gaussian Splatting.
• The README emphasizes high-performance rendering (∼30 FPS) on high-end hardware, support for advanced camera effects (defocus blur, distortion), and integrations with interactive remote viewers (SIBR).
Analyzing the File Structure
• Top-level files and directories:
• Multiple scripts for training (train.py), rendering (render.py), evaluation (metrics.py, full_eval.py, partial_eval.py), and hosting an interactive render server (host_render_server.py).
• Dedicated directories such as “arguments” (handling command line arguments), “gaussian_renderer” (likely containing rendering related modules including a special “ever.py”), “scene” (managing camera selection, dataset readers, and scene-related processing), “lpipsPyTorch” (for perceptual similarity measurements) and “utils” (providing general, camera, graphics, math, and system utilities).
• Other directories and submodules (e.g., “SIBR_viewers” for remote viewing and “submodules/simple-knn”) indicate that the system leverages well-separated components.
Key Components of the System and Their Relationships
a. Data Input & Dataset Processing:
• Dataset readers and loaders in the “scene” directory (colmap_loader.py, dataset_readers.py) manage input from COLMAP, NeRF Synthetic, and Zip-NeRF data.
• Scripts like "convert.py" and "extract_metadata.py" likely process or extract metadata from scene data.
b. Training Module:
• The training process is managed by train.py, which relies on modules from “scene”, “arguments”, and the “utils” packages.
• Training involves computing a volumetric model (note files such as scene/gaussian_model.py) using real-time differentiable volume rendering.
c. Rendering Pipeline:
• The render.py script, along with additional utilities, is focused on generating view synthesis renderings from a trained model.
• Canvas elements from the “gaussian_renderer” (such as fast_renderer.py and network_gui.py) are critical here.
d. Evaluation & Metrics:
• Evaluation workflows include metrics.py, full_eval.py, and partial_eval.py. These evaluate rendering errors and measure performance (like FPS with measure_fps.py) and perceptual quality (via lpipsPyTorch).
e. Interactive Viewing:
• The hosting of an interactive server via host_render_server.py for using SIBR remote viewers is another key component.
• The SIBR_viewers directory and provided instructions in the README show how the server connects with external viewer applications.
f. Supporting Utilities and Submodules:
• Generic utilities in the “utils” directory support camera operations (including specific variants like camera_utils_zipnerf.py), graphics, math, and system-level operations.
• The “arguments” directory and configuration scripts (like install.bash and run.sh) standardize the command-line setting and deployment aspects.
• The “submodules” folder indicates additional 3rd-party (or homemade) components like the simple-knn implementation, which might be used for some geometric processing.
Architectural Patterns and Technology Stack
• Architectural Patterns:
• Separation of concerns: distinct modules for training, rendering, evaluation and interactive viewing.
• A modular design that separates scene/data processing, rendering engine (gaussian_renderer), and client/server interactive viewer (using SIBR protocol).
• Command-line driven configuration using the “arguments” modules enhances flexibility.
• Technologies and Dependencies:
• The project uses NVIDIA’s OptiX for ray tracing, and SlangD for shader compilation.
• PyTorch is used for deep learning aspects (notably during training and evaluation).
• Additional libraries include OpenCV, GLFW, Eigen, and others as provided in the dependency list.
• The integration with SIBR indicates standardized viewer protocols/messaging.