How lpiccinelli-eth/unik3d works
Main Components
• Input & Preprocessing:
• External inputs: Single RGB images and camera parameter files (e.g., JSON files in the assets folder).
• Preprocessing logic applied on inputs – image normalization, resizing, and adapting intrinsics (documented in the README and usage code).
• Core UniK3D Library:
• Models:
• The “unik3d/models” directory shows the architecture as it provides the UniK3D model along with related submodules for encoder, decoder, and specialized backbone networks (e.g., ViT-based networks, ConvNeXt, etc.).
• Datasets:
• The “unik3d/datasets” package includes support for several dataset formats and pipelines, implying a modular dataset handling mechanism.
• Layers & Ops:
• “unik3d/layers” contains activation functions, attention modules, and other neural network layers (including novel blocks for metadinov2 and others).
• “unik3d/ops” provides custom operations, including specialized KNN functions (with CUDA) and defined losses, which are crucial for performance and efficiency.
• Utilities:
• “unik3d/utils” provides helper functions such as camera manipulations, evaluation metrics, geometric utilities, and visualization tools.
• Configuration and Scripts:
• Config files are provided in “configs/train” and “configs/eval” to parameterize different training and evaluation settings.
• Scripts in the “scripts” folder (demo, infer, eval, train) outline complete pipelines for inference/testing, guiding how data moves through the system.
• “app.py” and “gradio_demo.py” indicate that there are runnable demos for quick experimentation (possibly for a web interface with Gradio).
• External Interfaces:
• Integration with Hugging Face for model demo and pre-trained weights.
• TorchHub integration via “hubconf.py”, which allows users to load the models using PyTorch’s hub.
Establishing Relationships and Interactions
• External Input / User Interface: This includes the command-line demos (scripts), the Gradio demo, and the interactive Hugging Face Spaces.
• The UniK3D Library: Represent the internal modular structure subdivided into Models, Datasets, Layers/Ops, and Utils.
• Flow from the processed input into the Model (which resides in “unik3d/models”), then out to output predictions (depth, points, rays).
• A sub-flow where the configuration files (from “configs/”) parameterize the training or inference routines.
Architectural Patterns & Principles
• Modularity: The separation of datasets, models, and layers indicate a modular design where each component is interchangeable.
• Config-driven behavior: Use of configuration directories and scripts suggests a design that can shift parameters without code changes.
• Extensibility through plugins: The variety of dataset modules and backbones implies that people can extend the system by adding new dataset handlers or model variants.
• Hardware acceleration: The presence of CUDA-compiled operations in “unik3d/ops/knn” shows that performance optimizations are built into the design.
Detailed System Process
• API call from the demonstration code (or Gradio code) into the core library (specifically, using UniK3D.from_pretrained and then model.infer).
• Out the internal call stack from the inference function into utility functions (camera handling, evaluation) and model submodules (layers/ops used during the forward pass).