How zju3dv/street_gaussians works
Main Components
• Configuration Manager – This includes YAML configuration files in the “configs” folder and the Python modules found under lib/config. They define the parameters for training, rendering, and evaluation.
• Models and Rendering – The “lib/models” folder contains numerous components: different Gaussian models (e.g., actor, background, sky, street variants), camera pose handling, neural network GUI, and differentiable renderers. These pieces are interconnected, working together for scene synthesis and rendering.
• GPU-accelerated Submodules – The project uses submodules for performance-critical tasks:
– Diffusion Gaussian Rasterization is implemented in the “submodules/diff-gaussian-rasterization” directory. This portion includes both CUDA (.cu) files and CPU wrappers.
– Simple KNN – located under “submodules/simple-knn” and providing spatial querying and neighbor search functionality.
– Also, there is a submodule for the Waymo open-dataset reader for data extraction.
• Training and Rendering Scripts – Outside the modules, the user-facing scripts (e.g., train.py, render.py, make_ply.py) and shell scripts in “script/waymo” orchestrate the execution flow. They load configurations, invoke data pipelines, execute training loops, and trigger rendering.
• Visualization – After training and rendering, the system outputs formats (for instance, via make_ply.py) that can be viewed using external viewers (e.g., SIBR_viewers).
Relationships and Data Flow
• Data ingestion: Raw input (e.g., Waymo dataset, tracking predictions) is processed by conversion scripts. The dataset modules read this processed data and feed batches to the training modules.
• Training: The models (defined in lib/models) use processed dataset information along with auxiliary utilities (e.g., for loss calculation, camera geometry, etc., found in lib/utils) and GPU-accelerated operations from the diff-gaussian-rasterization submodule.
• Rendering: Once trained, the rendering pipeline uses a similar configuration path (render.py and accompanying scripts) to synthesize images from learned representations.
• Visualization: The output from rendering/training flows into post-processing (e.g., make_ply.py) for visualization and external viewer compatibility.
Important Architectural Patterns and Design Principles
• Modular Separation – Components are separated by function (data preprocessing, model definition, rendering, and utility libraries) making it easier to isolate and test modules.
• Pipeline Architecture – The system is built around sequential stages (data conversion → training → rendering → visualization) with scripts orchestrating transitions. This can be depicted as a linear pipeline with branching for sub-processes (e.g., multiple ways to preprocess or render).
• Hardware Acceleration – The use of dedicated CUDA-based submodules emphasizes the need to distinguish between CPU-bound logic and GPU-accelerated computation.
• Reusability and Extensibility – The clear separation into lib/ (core modules), script/ (entry point scripts), and submodules allows integration with external tools and reuse of components.
Technologies, Frameworks, and Libraries
• Python and PyTorch for model training and scripting.
• CUDA (C++/CUDA files in diff-gaussian-rasterization) for GPU-accelerated rendering.
• YAML for configuration.
• External libraries for image, graphics, and data utilities (as seen in lib/utils).
• Optional integration with external tools (SIBR_viewers) for visualization.
• Additional dependencies from the “requirements.txt” and installation instructions.