# lpiccinelli-eth/unidepth architecture

> Architecture diagram of the GitHub repository lpiccinelli-eth/unidepth, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lpiccinelli-eth/unidepth
- Repository: https://github.com/lpiccinelli-eth/unidepth

## Overview

##### PROJECT TYPE & PURPOSE
- Type: This is a deep learning computer vision project focused on depth estimation from monocular images.
- Purpose: It provides both training and inference pipelines for estimating metric depth from a single image using various neural network backbones (ConvNext and Vision Transformers) and incorporates custom CUDA operators (for example, for extracting patches and KNN routines).

##### TOP-LEVEL COMPONENTS (as per file structure)
- Assets and Configurations:
- The assets and configs directories hold example data (like demo images, intrinsics, and configurations files) and documentation.
- Scripts:
- Includes demo.py and train.py which guide inference/demonstration and training procedures, respectively.
- Package “unidepth”:
- Contains all core modules of the project including:
▪ datasets: Module to load and pre-process various depth datasets (both real and synthetic data).
▪ layers: Contains custom architectural building blocks (e.g., convolutions, attention, positional encoding).
▪ models: Includes specific implementations of UniDepthV1 and UniDepthV2 including corresponding decoders, backbones, and export utilities.
▪ ops: Implements custom CUDA/C++ operations needed for efficient processing (e.g., patch extraction, knn operations, and loss computations).
▪ utils: Provides auxiliary functionalities (e.g., camera transforms, evaluation metrics, visualization routines).
- Other files like hubconf.py and pyproject.toml indicate that the project is also set up to integrate with Torch Hub and supports packaging.

##### ARCHITECTURAL PATTERNS & DESIGN INSIGHTS
- Modular structure: The clear separation between datasets, models, operations, and utilities emphasizes a modular design that allows easy extension (e.g., adding new datasets or network backbones).
- Pipeline flow:
- Data ingestion and preprocessing handled in unidepth/datasets.
- Model architecture composed in unidepth/models (with backbones and decoders) and layers.
- Customized operations (accelerated with C++/CUDA) handled in unidepth/ops.
- Result post-processing, evaluation, and visualization conducted using the tools in unidepth/utils.
- Training and inference are orchestrated by top-level scripts (train.py and demo.py) and configured via JSON files in the configs directory.
- Extensibility: The project design also supports loading pre-trained weights from online repositories (e.g., Hugging Face) and even integration with external demos (e.g., Hugging Face/Gradio).

##### COMPONENT RELATIONSHIPS & DATA FLOW
- Input stage: An RGB image is loaded and pre-processed in a dataset module.
- Forward path (Inference/Training pipeline):
- The image is forwarded through the chosen backbone (e.g., ConvNext or a Vision Transformer variant) to extract features.
- A corresponding decoder model (from either UniDepthV1 or UniDepthV2) then produces depth estimations, predicted camera intrinsics, and point clouds.
- Custom operations (CUDA and CPU code) are incorporated for specialized tasks like patch extraction and efficient computation.
- Output stage:
- The final data (depth map, point cloud, camera intrinsics) is returned and optionally visualized.
- Training losses are computed using custom loss functions located in unidepth/ops/losses.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A("Input Image"):::data
    B("Data Ingestion & Preprocessing (Datasets)"):::data
    C("Backbone (Feature Extraction)"):::model
    L("Core Encoder Functionality"):::model
    D1("UniDepthV1 (Encoder/Decoder)"):::model
    D2("UniDepthV2 (Encoder/Decoder)"):::model
    F("Custom Layers & Architectural Blocks"):::model
    G("Custom Operations (CUDA/C++ Ops)"):::ops
    H("Post-processing & Utilities"):::utils
    I1("Training Pipeline (train.py)"):::scripts
    I2("Inference/Demo (demo.py)"):::scripts
    J1("Assets (Demo, Docs)"):::external
    J2("Configuration Files"):::external
    K("External Integrations (Hugging Face/TorchHub)"):::external

    %% Data Flow
    A --> B
    B --> C
    C --> L
    L --> D1
    L --> D2
    F --> D1
    F --> D2
    G --> D1
    G --> D2
    D1 --> H
    D2 --> H

    %% Scripts triggering the pipeline
    I1 --> B
    I1 --> D1
    I1 --> D2
    I2 --> B
    I2 --> D1
    I2 --> D2
    I2 --> H
    H -- "loss_computation" --> I1

    %% External assets and configurations supply data and parameters
    J1 --> B
    J1 --> I2
    J2 --> B
    J2 --> I1
    J2 --> I2

    %% External integration for pre-trained weights and demos
    K --> D1
    K --> D2

    %% Click Events
    click B "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/datasets"
    click C "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/models/backbones"
    click L "https://github.com/lpiccinelli-eth/unidepth/blob/main/unidepth/models/encoder.py"
    click D1 "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/models/unidepthv1"
    click D2 "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/models/unidepthv2"
    click F "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/layers"
    click G "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/ops"
    click H "https://github.com/lpiccinelli-eth/unidepth/tree/main/unidepth/utils"
    click I1 "https://github.com/lpiccinelli-eth/unidepth/blob/main/scripts/train.py"
    click I2 "https://github.com/lpiccinelli-eth/unidepth/blob/main/scripts/demo.py"
    click J1 "https://github.com/lpiccinelli-eth/unidepth/tree/main/assets"
    click J2 "https://github.com/lpiccinelli-eth/unidepth/tree/main/configs"

    %% Styles
    classDef data fill:#B3E5FC,stroke:#0277BD,color:#000,stroke-width:2px;
    classDef model fill:#C8E6C9,stroke:#2E7D32,color:#000,stroke-width:2px;
    classDef ops fill:#FFCDD2,stroke:#C62828,color:#000,stroke-width:2px;
    classDef utils fill:#FFE0B2,stroke:#EF6C00,color:#000,stroke-width:2px;
    classDef scripts fill:#D1C4E9,stroke:#5E35B1,color:#000,stroke-width:2px;
    classDef external fill:#F0F4C3,stroke:#827717,color:#000,stroke-width:2px;
```
