# bytedance-seed/depth-anything-3 architecture

> Architecture diagram of the GitHub repository bytedance-seed/depth-anything-3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance-seed/depth-anything-3
- Repository: https://github.com/bytedance-seed/depth-anything-3

## Overview

1. **Repository shape**
   - This is primarily an **inference and evaluation codebase** for Depth Anything 3, not a training repo.
   - It is split into two major packages:
     - `src/depth_anything_3` for the main model, API, CLI, UI, export, and benchmark tooling.
     - `da3_streaming` for long-sequence / sliding-window streaming inference and loop-closure refinement.

2. **Core model boundary**
   - The central runtime object is `DepthAnything3` in `src/depth_anything_3/api.py`.
   - It acts as the main façade over model loading, input normalization, batching, inference, and result packaging.
   - Underneath it, the actual neural network is `DepthAnything3Net` in `src/depth_anything_3/model/da3.py`.

3. **Model composition**
   - The network is assembled from config-defined components:
     - a **DINOv2-based transformer backbone** in `model/dinov2/`
     - a **DPT-style decoder/head** family in `model/dpt.py`, `dualdpt.py`, `gsdpt.py`, `cam_dec.py`, `cam_enc.py`
     - optional adapters for **camera pose** and **3D Gaussian** outputs
   - This architecture reflects the README claim that one plain transformer plus a unified depth-ray representation can support multiple tasks.

4. **Configuration and dynamic instantiation**
   - Model variants are defined in YAML under `src/depth_anything_3/configs/`.
   - `cfg.py`, `registry.py`, and `utils/registry.py` support config-driven object creation via `create_object` / `load_config`.
   - This is a key architecture boundary: model behavior is selected by configuration rather than hard-coded subclasses.

5. **Input path**
   - Inputs are handled through `utils/io/input_processor.py`, `services/input_handlers.py`, and the app-side file handlers.
   - The system accepts images, folders, and video inputs, with optional reference-view selection for multi-view scenarios.
   - In the streaming package, input is processed in sliding windows to keep GPU memory bounded for very long video sequences.

6. **Inference and orchestration**
   - `services/inference_service.py` and `services/backend.py` coordinate reusable model execution for the app and CLI.
   - The backend can be kept resident on GPU for repeated jobs, which is why the README exposes `da3 backend ... --use-backend`.
   - This backend reuse is a distinct boundary from one-shot CLI execution.

7. **Output and post-processing**
   - Results are packaged through `utils/io/output_processor.py` and related helper modules.
   - The main outputs are:
     - processed RGB frames
     - depth maps
     - confidence maps
     - camera intrinsics / extrinsics
   - Additional geometry utilities handle pose alignment, camera trajectory conversion, and ray/scene math.

8. **Export pipeline**
   - Export is a major subsystem under `utils/export/`.
   - Supported outputs include `glb`, `npz`, depth visualizations, feature visualizations, COLMAP-style assets, and Gaussian-splatting-related formats.
   - This export layer is the main boundary between model inference and downstream 3D tools.

9. **Interactive app layer**
   - The Gradio UI lives in `src/depth_anything_3/app/`.
   - It is split into HTML/CSS, UI components, event handlers, file handlers, inference hooks, and visualization utilities.
   - The app is a thin orchestration layer over the same backend/model pipeline used by the CLI.

10. **CLI and operational modes**
    - `cli.py` exposes the command-line entry points documented in `docs/CLI.md`.
    - The CLI supports at least three operational patterns visible in the README:
      - backend-only caching
      - auto processing over image sets
      - video processing with export and feature visualization
    - The same inference core is reused across these modes, which keeps behavior consistent between UI and batch workflows.

11. **Benchmark subsystem**
    - `src/depth_anything_3/bench/` is a separate evaluation pipeline with dataset adapters, evaluator logic, registries, and metric reporting.
    - Dataset support includes DTU, ETH3D, HiRoom, ScanNet++, and 7Scenes.
    - This part of the repo is for standardized evaluation of pose estimation, reconstruction, and rendering quality rather than inference serving.

12. **Streaming extension**
    - `da3_streaming/` extends DA3 for ultra-long sequence inference.
    - It contains:
      - streaming configs
      - a fast loop-closure solver in C++ (`fastloop/solve.cpp`) with a Python wrapper
      - loop detection, alignment, refinement, and Sim3 utilities
    - The architecture here is a memory-efficient sliding-window pipeline with geometric correction across windows.

13. **External dependencies that matter architecturally**
    - The repo relies on PyTorch for inference, Gradio for the UI, and Hugging Face model hosting for pretrained weights.
    - `gsplat` is an optional dependency for Gaussian-head functionality.
    - Triton appears in the alignment utilities, suggesting GPU-accelerated alternatives for some geometric kernels.

14. **Practical architecture summary**
    - At a high level, the system is: **input ingestion → config-driven model inference → geometric post-processing → export/visualization/evaluation**.
    - The major boundaries are between:
      - frontend surfaces (CLI / Gradio / streaming)
      - the reusable inference backend
      - the model implementation
      - export and benchmark consumers
      - the streaming-specific long-sequence correction pipeline

Main entry points: [`src/depth_anything_3/model/da3.py`](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/da3.py) (DA3 net), [`src/depth_anything_3/api.py`](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/api.py) (DA3 API), [`src/depth_anything_3/services/inference_service.py`](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/inference_service.py) (Inference svc), [`da3_streaming/da3_streaming.py`](https://github.com/bytedance-seed/depth-anything-3/blob/main/da3_streaming/da3_streaming.py) (Streaming)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_frontends["Surfaces"]
  node_cli["CLI<br/>entry point<br/>[cli.py]"]
  node_gradio_app["Gradio app<br/>ui<br/>[gradio_app.py]"]
  node_backend["Backend<br/>resident service<br/>[backend.py]"]
end

subgraph group_g_core["Core inference"]
  node_inference_service["Inference svc<br/>orchestrator"]
  node_input_handlers["Input handlers<br/>ingestion<br/>[input_handlers.py]"]
  node_input_processor["Input processor<br/>preprocess<br/>[input_processor.py]"]
  node_da3_api["DA3 API<br/>façade<br/>[api.py]"]
end

subgraph group_g_model["Model stack"]
  node_da3_net["DA3 net<br/>model runtime<br/>[da3.py]"]
  node_cfg_registry["Config registry<br/>config system<br/>[cfg.py]"]
  node_backbone["DINOv2<br/>vision backbone<br/>[dinov2.py]"]
  node_decoder_family["Decoders<br/>task heads<br/>[dpt.py]"]
  node_camera_heads["Camera heads<br/>pose heads<br/>[cam_enc.py]"]
  node_gs_adapter["GS adapter<br/>gaussian bridge<br/>[gs_adapter.py]"]
end

subgraph group_g_post["Post-process"]
  node_output_processor["Output processor<br/>packaging"]
  node_geometry["Geometry<br/>math utils<br/>[geometry.py]"]
  node_export["Export<br/>downstream I/O"]
end

subgraph group_g_bench["Evaluation"]
  node_bench["Bench runner<br/>evaluation pipeline<br/>[evaluator.py]"]
  node_bench_datasets["Datasets<br/>dataset adapters"]
end

subgraph group_g_stream["Streaming"]
  node_streaming["Streaming<br/>long-sequence pipeline<br/>[da3_streaming.py]"]
  node_loop_utils["Loop utils<br/>geometry refinement"]
  node_fastloop["Fast loop<br/>native solver<br/>[solve.cpp]"]
end

node_cli -->|"uses"| node_inference_service
node_gradio_app -->|"uses"| node_inference_service
node_backend -->|"hosts"| node_da3_api
node_inference_service -->|"ingests"| node_input_handlers
node_input_handlers -->|"prepares"| node_input_processor
node_inference_service -->|"runs"| node_da3_api
node_da3_api -->|"loads"| node_cfg_registry
node_cfg_registry -->|"constructs"| node_da3_net
node_da3_net -->|"extracts features"| node_backbone
node_da3_net -->|"decodes"| node_decoder_family
node_da3_net -->|"predicts"| node_camera_heads
node_da3_net -.->|"supports"| node_gs_adapter
node_da3_api -->|"packages"| node_output_processor
node_output_processor -->|"uses"| node_geometry
node_output_processor -->|"exports"| node_export
node_bench -->|"evaluates"| node_bench_datasets
node_streaming -->|"reuses"| node_da3_api
node_streaming -->|"refines"| node_loop_utils
node_loop_utils -->|"accelerates"| node_fastloop
node_streaming -->|"windows"| node_input_processor

click node_cli "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/cli.py"
click node_gradio_app "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/app/gradio_app.py"
click node_backend "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/backend.py"
click node_inference_service "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/inference_service.py"
click node_input_handlers "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/input_handlers.py"
click node_input_processor "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/utils/io/input_processor.py"
click node_output_processor "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/utils/io/output_processor.py"
click node_geometry "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/utils/geometry.py"
click node_export "https://github.com/bytedance-seed/depth-anything-3/tree/main/src/depth_anything_3/utils/export"
click node_da3_api "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/api.py"
click node_da3_net "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/da3.py"
click node_cfg_registry "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/cfg.py"
click node_backbone "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/dinov2/dinov2.py"
click node_decoder_family "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/dpt.py"
click node_camera_heads "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/cam_enc.py"
click node_gs_adapter "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/gs_adapter.py"
click node_bench "https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/bench/evaluator.py"
click node_bench_datasets "https://github.com/bytedance-seed/depth-anything-3/tree/main/src/depth_anything_3/bench/datasets"
click node_streaming "https://github.com/bytedance-seed/depth-anything-3/blob/main/da3_streaming/da3_streaming.py"
click node_loop_utils "https://github.com/bytedance-seed/depth-anything-3/tree/main/da3_streaming/loop_utils"
click node_fastloop "https://github.com/bytedance-seed/depth-anything-3/blob/main/da3_streaming/fastloop/solve.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_gradio_app,node_backend toneBlue
class node_inference_service,node_input_handlers,node_input_processor,node_da3_api toneAmber
class node_da3_net,node_cfg_registry,node_backbone,node_decoder_family,node_camera_heads,node_gs_adapter toneMint
class node_output_processor,node_geometry,node_export toneRose
class node_bench,node_bench_datasets toneIndigo
class node_streaming,node_loop_utils,node_fastloop toneTeal
```

## Components

### Surfaces

CLI, app, and reusable backend entry points.

- [CLI](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/cli.py): entry point · Command-line workflows for backend reuse, batch processing, and video inference. · `src/depth_anything_3/cli.py`
- [Gradio app](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/app/gradio_app.py): ui · Interactive web UI built on the shared inference services. · `src/depth_anything_3/app/gradio_app.py`
- [Backend](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/backend.py): resident service · Keeps model execution resident for repeated jobs and app reuse. · `src/depth_anything_3/services/backend.py`

### Core inference

Shared runtime for loading models, processing inputs, and running inference.

- [Inference svc](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/inference_service.py): orchestrator · Coordinates reusable inference flows across UI and CLI surfaces. · `src/depth_anything_3/services/inference_service.py`
- [Input handlers](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/services/input_handlers.py): ingestion · Handles images, folders, videos, and reference-view selection. · `src/depth_anything_3/services/input_handlers.py`
- [Input processor](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/utils/io/input_processor.py): preprocess · Normalizes and batches inputs before model execution. · `src/depth_anything_3/utils/io/input_processor.py`
- [DA3 API](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/api.py): façade · Main runtime façade over loading, normalization, inference, and packaging. · `src/depth_anything_3/api.py`

### Model stack

Config-driven network assembly and task-specific heads.

- [DA3 net](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/da3.py): model runtime · Neural network implementation behind the public API. · `src/depth_anything_3/model/da3.py`
- [Config registry](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/cfg.py): config system · Loads YAML configs and instantiates objects dynamically. · `src/depth_anything_3/cfg.py`
- [DINOv2](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/dinov2/dinov2.py): vision backbone · Transformer backbone used as the shared feature extractor. · `src/depth_anything_3/model/dinov2/dinov2.py`
- [Decoders](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/dpt.py): task heads · DPT-style decoder family for depth, dual, metric, and nested variants. · `src/depth_anything_3/model/dpt.py`
- [Camera heads](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/cam_enc.py): pose heads · Camera encoding and decoding adapters for pose-related outputs. · `src/depth_anything_3/model/cam_enc.py`
- [GS adapter](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/model/gs_adapter.py): gaussian bridge · Optional Gaussian-splatting path and renderer integration. · `src/depth_anything_3/model/gs_adapter.py`

### Post-process

Geometry, output packaging, and export to downstream formats.

- [Output processor](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/utils/io/output_processor.py): packaging · Packages processed frames, depth, confidence, and camera outputs. · `src/depth_anything_3/utils/io/output_processor.py`
- [Geometry](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/utils/geometry.py): math utils · Shared ray, pose, and scene geometry helpers. · `src/depth_anything_3/utils/geometry.py`
- [Export](https://github.com/bytedance-seed/depth-anything-3/tree/main/src/depth_anything_3/utils/export): downstream I/O · Exports inference results to visualization and 3D tool formats. · `src/depth_anything_3/utils/export`

### Evaluation

Dataset adapters and benchmark execution.

- [Bench runner](https://github.com/bytedance-seed/depth-anything-3/blob/main/src/depth_anything_3/bench/evaluator.py): evaluation pipeline · Coordinates standardized evaluation and metric reporting. · `src/depth_anything_3/bench/evaluator.py`
- [Datasets](https://github.com/bytedance-seed/depth-anything-3/tree/main/src/depth_anything_3/bench/datasets): dataset adapters · Adapters for DTU, ETH3D, HiRoom, ScanNet++, and 7Scenes. · `src/depth_anything_3/bench/datasets`

### Streaming

Long-sequence sliding-window inference and loop refinement.

- [Streaming](https://github.com/bytedance-seed/depth-anything-3/blob/main/da3_streaming/da3_streaming.py): long-sequence pipeline · Sliding-window inference for long videos and sequence-level correction. · `da3_streaming/da3_streaming.py`
- [Loop utils](https://github.com/bytedance-seed/depth-anything-3/tree/main/da3_streaming/loop_utils): geometry refinement · Loop detection, alignment, Sim3 utilities, and refinement logic. · `da3_streaming/loop_utils`
- [Fast loop](https://github.com/bytedance-seed/depth-anything-3/blob/main/da3_streaming/fastloop/solve.cpp): native solver · C++ solver and Python wrapper for fast loop-closure optimization. · `da3_streaming/fastloop/solve.cpp`

## Connections

- CLI → Inference svc: uses · CLI routes batch and video workflows through the shared service layer.
- Gradio app → Inference svc: uses · UI events call the same inference orchestration as the CLI.
- Backend → DA3 API: hosts · Resident backend keeps the model façade loaded for repeated jobs.
- Inference svc → Input handlers: ingests · Selects and prepares images, folders, videos, and reference views.
- Input handlers → Input processor: prepares · Turns user inputs into normalized batches and runtime-ready tensors.
- Inference svc → DA3 API: runs · Executes inference through the central façade.
- DA3 API → Config registry: loads · Resolves model variants and runtime objects from config.
- Config registry → DA3 net: constructs · Builds the underlying network from config-defined components.
- DA3 net → DINOv2: extracts features · Uses DINOv2 as the shared transformer encoder.
- DA3 net → Decoders: decodes · Applies DPT-style and variant-specific heads.
- DA3 net → Camera heads: predicts · Optionally attaches camera pose and camera-encoding branches.
- DA3 net → GS adapter: supports · Routes optional Gaussian-splatting outputs through adapter logic.
- DA3 API → Output processor: packages · Formats model results into frames, maps, confidence, and cameras.
- Output processor → Geometry: uses · Relies on geometry helpers for pose alignment and scene math.
- Output processor → Export: exports · Sends results to GLB, NPZ, COLMAP, depth, feature, and GS outputs.
- Bench runner → Datasets: evaluates · Runs standardized benchmarks over dataset-specific adapters.
- Streaming → DA3 API: reuses · Streaming inference reuses the same model façade where possible.
- Streaming → Loop utils: refines · Applies loop detection, alignment, and Sim3 correction across windows.
- Loop utils → Fast loop: accelerates · Uses the native solver for fast loop-closure optimization.
- Streaming → Input processor: windows · Processes long videos in sliding windows to bound memory usage.