# isl-org/dpt architecture

> Architecture diagram of the GitHub repository isl-org/dpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/isl-org/dpt
- Repository: https://github.com/isl-org/dpt

## Overview

##### Project type and purpose
- Project type: A Python-based inference library and command-line tool for dense prediction tasks (monocular depth estimation and semantic segmentation) based on Vision Transformers.
- Purpose: Load pretrained DPT models, preprocess input images, run forward passes through transformer backbones plus task-specific heads, postprocess predictions, and write outputs to disk.

##### File structure
- Top-level scripts:
– run_monodepth.py
– run_segmentation.py
These are the CLI entry points.
- Core library: dpt/
– vit.py: Vision Transformer backbone
– blocks.py, base_model.py: building blocks and abstract model class
– midas_net.py, models.py: model definitions combining backbone + head
– transforms.py: input preprocessing (resize, normalization)
- Utility library: util/
– io.py: image I/O (read, write)
– misc.py: helper functions
- Asset folders:
– weights/: pretrained .pt files
– input/: placeholder for input images
– output_monodepth/, output_semseg/: result directories
- Config/deployment:
– requirements.txt, setup.py: Python packaging and dependencies
– .gitignore, LICENSE, README.md

##### README for additional insights
- External dependencies: Python 3.7+, PyTorch 1.8.0, OpenCV 4.5.1, timm 0.4.5
- Deployment flow: download weights → install dependencies → place images in input/ → run CLI script → outputs in respective folders

##### Core components and data flow
a. Input Layer
– CLI script reads file paths from input/
– util/io loads images via OpenCV
b. Preprocessing
– dpt.transforms applies resizing, normalization
c. Model Loader
– dpt.models or midas_net instantiates the network architecture
– weights loaded via PyTorch’s `load_state_dict`
d. Inference Engine
– Forward pass: vit backbone → feature maps → task head (depth or segmentation)
e. Postprocessing
– Depth: scale or normalize output tensor → save as grayscale image
f. Output Layer
– util/io writes files to output_monodepth/ or output_semseg/

##### Architectural patterns and design principles
- Modular pipeline pattern: distinct stages (I/O → preprocess → model → postprocess)
- Separation of concerns: dpt/ for model logic, util/ for I/O and visualization
- CLI orchestration: top-level scripts glue together modules without hardcoding details

##### Putting it all together
– CLI Layer (run_monodepth.py, run_segmentation.py)
– Input Directory and util/io
– dpt.transforms
– dpt model definitions (vit backbone + head)
– Weights store
– util/pallete (for segmentation)
– Output directories

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Layer"
        Monodepth["run_monodepth.py"]:::cli
        Segmentation["run_segmentation.py"]:::cli
    end

    %% Input & I/O
    InputDir["Input Directory<br/>(raw images)"]:::storage
    IO["util/io.py<br/>Image I/O"]:::util

    %% Preprocessing
    Transform["dpt/transforms.py<br/>Preprocessing"]:::model

    %% Model Components
    subgraph "Model Components"
        Backbone["dpt/vit.py<br/>ViT Backbone"]:::model
        Blocks["dpt/blocks.py<br/>Building Blocks"]:::model
        BaseModel["dpt/base_model.py<br/>Abstract Model"]:::model
        Models["dpt/models.py<br/>Model Definitions"]:::model
        MiDaS["dpt/midas_net.py<br/>MiDaS Wrapper"]:::model
    end

    %% Postprocessing & Output
    Palette["util/pallete.py<br/>Color Mapping"]:::util
    Misc["util/misc.py<br/>Helper Utils"]:::util
    OutputDepth["output_monodepth/"]:::storage
    OutputSeg["output_semseg/"]:::storage

    %% Weights Store
    Weights["Weights Store<br/>(.pt files)"]:::storage

    %% External Libraries
    subgraph "External Libraries"
        PyTorch["PyTorch"]:::external
        TIMM["timm"]:::external
        OpenCV["OpenCV"]:::external
    end

    %% Data Flow
    Monodepth -->|reads paths| InputDir
    Segmentation -->|reads paths| InputDir
    InputDir -->|image files| IO
    IO -->|raw images| Transform
    Transform -->|tensors| Backbone
    Backbone --> Models
    Blocks --> Models
    BaseModel --> Models
    Models --> MiDaS
    Weights --> Models
    PyTorch --> Models
    TIMM --> Backbone
    OpenCV --> IO

    MiDaS -->|depth tensor| IO
    IO -->|writes files| OutputDepth

    MiDaS -->|seg indices| Palette
    Palette -->|color images| IO
    IO -->|writes files| OutputSeg

    %% Click Events
    click Monodepth "https://github.com/isl-org/dpt/blob/main/run_monodepth.py"
    click Segmentation "https://github.com/isl-org/dpt/blob/main/run_segmentation.py"
    click InputDir "https://github.com/isl-org/dpt/tree/main/input/"
    click IO "https://github.com/isl-org/dpt/blob/main/util/io.py"
    click Transform "https://github.com/isl-org/dpt/blob/main/dpt/transforms.py"
    click Backbone "https://github.com/isl-org/dpt/blob/main/dpt/vit.py"
    click Blocks "https://github.com/isl-org/dpt/blob/main/dpt/blocks.py"
    click BaseModel "https://github.com/isl-org/dpt/blob/main/dpt/base_model.py"
    click Models "https://github.com/isl-org/dpt/blob/main/dpt/models.py"
    click MiDaS "https://github.com/isl-org/dpt/blob/main/dpt/midas_net.py"
    click Palette "https://github.com/isl-org/dpt/blob/main/util/pallete.py"
    click Misc "https://github.com/isl-org/dpt/blob/main/util/misc.py"
    click Weights "https://github.com/isl-org/dpt/tree/main/weights/"
    click OutputDepth "https://github.com/isl-org/dpt/tree/main/output_monodepth/"
    click OutputSeg "https://github.com/isl-org/dpt/tree/main/output_semseg/"

    %% Styles
    classDef cli fill:#D6E9FC,stroke:#4A90E2;
    classDef util fill:#E8F5E9,stroke:#388E3C;
    classDef model fill:#FFF9C4,stroke:#FBC02D;
    classDef storage fill:#F5F5F5,stroke:#9E9E9E;
    classDef external fill:#FCE4EC,stroke:#D81B60;
```
