# lpiccinelli-eth/unik3d architecture

> Architecture diagram of the GitHub repository lpiccinelli-eth/unik3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lpiccinelli-eth/unik3d
- Repository: https://github.com/lpiccinelli-eth/unik3d

## Overview

##### Main Components
- Input & Preprocessing:
- External inputs: Single RGB images and camera parameter files (e.g., JSON files in the assets folder).
- Preprocessing logic applied on inputs – image normalization, resizing, and adapting intrinsics (documented in the README and usage code).

- Core UniK3D Library:
- Models:
- The “unik3d/models” directory shows the architecture as it provides the UniK3D model along with related submodules for encoder, decoder, and specialized backbone networks (e.g., ViT-based networks, ConvNeXt, etc.).
- Datasets:
- The “unik3d/datasets” package includes support for several dataset formats and pipelines, implying a modular dataset handling mechanism.
- Layers & Ops:
- “unik3d/layers” contains activation functions, attention modules, and other neural network layers (including novel blocks for metadinov2 and others).
- “unik3d/ops” provides custom operations, including specialized KNN functions (with CUDA) and defined losses, which are crucial for performance and efficiency.
- Utilities:
- “unik3d/utils” provides helper functions such as camera manipulations, evaluation metrics, geometric utilities, and visualization tools.

- Configuration and Scripts:
- Config files are provided in “configs/train” and “configs/eval” to parameterize different training and evaluation settings.
- Scripts in the “scripts” folder (demo, infer, eval, train) outline complete pipelines for inference/testing, guiding how data moves through the system.
- “app.py” and “gradio_demo.py” indicate that there are runnable demos for quick experimentation (possibly for a web interface with Gradio).

- External Interfaces:
- Integration with Hugging Face for model demo and pre-trained weights.
- TorchHub integration via “hubconf.py”, which allows users to load the models using PyTorch’s hub.

##### Establishing Relationships and Interactions
- External Input / User Interface: This includes the command-line demos (scripts), the Gradio demo, and the interactive Hugging Face Spaces.
- The UniK3D Library: Represent the internal modular structure subdivided into Models, Datasets, Layers/Ops, and Utils.
- Flow from the processed input into the Model (which resides in “unik3d/models”), then out to output predictions (depth, points, rays).
- A sub-flow where the configuration files (from “configs/”) parameterize the training or inference routines.

##### Architectural Patterns & Principles
- Modularity: The separation of datasets, models, and layers indicate a modular design where each component is interchangeable.
- Config-driven behavior: Use of configuration directories and scripts suggests a design that can shift parameters without code changes.
- Extensibility through plugins: The variety of dataset modules and backbones implies that people can extend the system by adding new dataset handlers or model variants.
- Hardware acceleration: The presence of CUDA-compiled operations in “unik3d/ops/knn” shows that performance optimizations are built into the design.

##### Detailed System Process
- API call from the demonstration code (or Gradio code) into the core library (specifically, using UniK3D.from_pretrained and then model.infer).
- Out the internal call stack from the inference function into utility functions (camera handling, evaluation) and model submodules (layers/ops used during the forward pass).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Input and Preprocessing Layer
    subgraph "Input & Preprocessing"
        EI("External Inputs"):::external
        PP("Preprocessing Module"):::external
    end

    %% Core UniK3D Library
    subgraph "Core UniK3D Library"
        M("Models"):::core
        DH("Datasets Handler"):::core
        L("Layers"):::core
        O("Ops"):::core
        U("Utilities"):::core
    end

    %% Configuration and Scripts
    subgraph "Configuration and Scripts"
        CF("Configuration Files"):::config
        PS("Pipeline Scripts"):::config
        A("app.py"):::config
        GD("gradio_demo.py"):::config
    end

    %% External Interfaces
    subgraph "External Interfaces"
        TH("TorchHub Integration"):::config
    end

    %% Inference Outputs
    IO("Inference Outputs<br/>(Depth, 3D Points, Rays)"):::core

    %% Connections
    EI -->|"feeds"| PP
    PP -->|"processed input"| M
    M -->|"predicts"| IO

    %% Internal dependencies within Core Library
    M ---|"utilizes"| DH
    M ---|"utilizes"| L
    M ---|"utilizes"| O
    M ---|"utilizes"| U

    %% Configuration influences
    CF ---|"configures"| M
    PS ---|"orchestrates"| PP
    TH ---|"integrates"| M

    %% Styles
    classDef external fill:#ccffcc,stroke:#333,stroke-width:2px
    classDef core fill:#cce5ff,stroke:#333,stroke-width:2px
    classDef config fill:#ffe5cc,stroke:#333,stroke-width:2px

    %% Click Events
    click EI "https://github.com/lpiccinelli-eth/unik3d/blob/main/assets/demo/dl3dv.json"
    click PP "https://github.com/lpiccinelli-eth/unik3d/blob/main/unik3d/datasets/pipelines/transforms.py"
    click M "https://github.com/lpiccinelli-eth/unik3d/blob/main/unik3d/models/unik3d.py"
    click DH "https://github.com/lpiccinelli-eth/unik3d/tree/main/unik3d/datasets"
    click L "https://github.com/lpiccinelli-eth/unik3d/tree/main/unik3d/layers"
    click O "https://github.com/lpiccinelli-eth/unik3d/tree/main/unik3d/ops"
    click U "https://github.com/lpiccinelli-eth/unik3d/tree/main/unik3d/utils"
    click CF "https://github.com/lpiccinelli-eth/unik3d/tree/main/configs/train"
    click PS "https://github.com/lpiccinelli-eth/unik3d/tree/main/scripts"
    click A "https://github.com/lpiccinelli-eth/unik3d/blob/main/app.py"
    click GD "https://github.com/lpiccinelli-eth/unik3d/blob/main/gradio_demo.py"
    click TH "https://github.com/lpiccinelli-eth/unik3d/blob/main/hubconf.py"
```
