How srameo/le3d works
Main Components
• Data & Preprocessing:
– Represent the raw image input and metadata extraction pipelines.
– Highlight the “basicgs/data” directory (which contains utilities such as camera_utils, colmap_dataset, colmap_utils, and prefetch_dataloader) and the “datasets” folder.
– Include scripts such as “scripts/data/extract_exif_as_json.sh” and “scripts/data/local_colmap.sh” for exif extraction and COLMAP-based camera calibration.
• 3D Gaussian Modules:
– Represent the core 3D Gaussian Splatting engine. This lives under “basicgs/gaussians”.
– Show submodules inside “basicgs/gaussians/util” (e.g., graphics, op, ply, render, sh, transform) and highlight that they implement the mathematical and rendering operations needed for 3D splatting.
– Include the different architectural variations such as “color_mlp_arch.py” and “le3d_arch.py.”
• Losses & Metrics:
• Pipelines & Training:
– Show the training flows via “basicgs/pipelines” (which includes pipelines such as “base_pipeline.py”, “gs_base_pipeline.py”, “le3d_pipeline.py” and a learning rate scheduler).
– Highlight the entry points “basicgs/train.py” and “basicgs/test.py” as scripts that start the training and evaluation.
• Utilities & Configuration:
– Note the “options” directory, where YAML configuration files (for different scenes or variants like “le3d” and “vanilla_gs”) are used to customize training/execution.
• Viewing & Editing:
– Represent the interactive components in “basicgs/view” and “basicgs/viewers”.
– Include both the base viewer and specialized editors (e.g., “editor.py”, “le3d_viewer.py”) and their associated utilities (like export_util, frame_util, interpolate_util).
– Connect these components to the post-training export tool “scripts/export_splat.sh” which converts output point cloud (.ply) files to splat files for the web viewer.
• External/3rd-Party Submodules:
– Highlight the “submodules” directory. It includes independent, externally maintained libraries for different aspects of rasterization (like base-rasterization, full-rasterization, gcnt-rasterization, hdr-splat, and hist-rasterization). These are key for achieving GPU-accelerated rendering.
– Also note that this project borrows code from BasicSR and gaussian-splatting projects, so indicate these as external dependencies that integrate with the core system.
Relationships and Interaction Flows
• Data Pipeline Flow:
– Raw input images and DNG files are first processed using the data utilities and scripts to extract metadata and calibrate camera positions.
– The processed data flows into the training pipeline.
• Training Flow:
– The training pipeline (configured via YAML files in “options”) loads data, builds models using modules defined in “basicgs/gaussians”, applies loss functions (from “basicgs/losses”), and computes metrics (from “basicgs/metrics”).
– Utilities help manage options and logging.
• Rendering & Viewing Flow:
– Once training is complete, a point cloud (PLY file) is generated.
– This is then converted into a rendered output (splat file) using provided export scripts.
• External Rendering Submodules:
– The GPU-accelerated rasterization submodules integrate with the core code to perform real-time display and render computations.
Architectural Patterns and Design Principles
• Modularity: The code is organized into discrete modules (data, gaussians, losses, pipelines, viewers, utils) to separate concerns.
• Configuration-driven: The “options” directory and YAML files drive many aspects of training and inference.
• Extensibility: The use of registries (such as in “basicgs/utils/registry.py”) suggests a plugin-like architecture that enables easy extension.
• Real-Time Rendering Considerations: Use of GPU submodules for rasterization and CUDA kernels in submodules indicates optimized real-time rendering.
Relevant Technologies and Frameworks
• Python as the main programming language.
• Pytorch for deep learning and training the 3D Gaussian models.
• CUDA for GPU acceleration (with scripts for installing CUDA support and inferred use of custom CUDA kernels in submodules).
• COLMAP for camera calibration and possibly for sparse reconstruction.
• Integration with BasicSR style framework and reliance on external libraries from “submodules” for advanced rasterization.