How bytedance-seed/depth-anything-3 works
1. Repository shape
- This is primarily an inference and evaluation codebase for Depth Anything 3, not a training repo.
- It is split into two major packages:
- src/depth_anything_3 for the main model, API, CLI, UI, export, and benchmark tooling.
- da3_streaming for long-sequence / sliding-window streaming inference and loop-closure refinement.
2. Core model boundary
- The central runtime object is DepthAnything3 in src/depth_anything_3/api.py.
- It acts as the main façade over model loading, input normalization, batching, inference, and result packaging.
- Underneath it, the actual neural network is DepthAnything3Net in src/depth_anything_3/model/da3.py.
3. Model composition
- The network is assembled from config-defined components:
- a DINOv2-based transformer backbone in model/dinov2/
- a DPT-style decoder/head family in model/dpt.py, dualdpt.py, gsdpt.py, cam_dec.py, cam_enc.py
- optional adapters for camera pose and 3D Gaussian outputs
- This architecture reflects the README claim that one plain transformer plus a unified depth-ray representation can support multiple tasks.
4. Configuration and dynamic instantiation
- Model variants are defined in YAML under src/depth_anything_3/configs/.
- cfg.py, registry.py, and utils/registry.py support config-driven object creation via create_object / load_config.
- This is a key architecture boundary: model behavior is selected by configuration rather than hard-coded subclasses.
5. Input path
- Inputs are handled through utils/io/input_processor.py, services/input_handlers.py, and the app-side file handlers.
- The system accepts images, folders, and video inputs, with optional reference-view selection for multi-view scenarios.
- In the streaming package, input is processed in sliding windows to keep GPU memory bounded for very long video sequences.
6. Inference and orchestration
- services/inference_service.py and services/backend.py coordinate reusable model execution for the app and CLI.
- The backend can be kept resident on GPU for repeated jobs, which is why the README exposes da3 backend ... --use-backend.
- This backend reuse is a distinct boundary from one-shot CLI execution.
7. Output and post-processing
- Results are packaged through utils/io/output_processor.py and related helper modules.
- The main outputs are:
- processed RGB frames
- depth maps
- confidence maps
- camera intrinsics / extrinsics
- Additional geometry utilities handle pose alignment, camera trajectory conversion, and ray/scene math.
8. Export pipeline
- Export is a major subsystem under utils/export/.
- Supported outputs include glb, npz, depth visualizations, feature visualizations, COLMAP-style assets, and Gaussian-splatting-related formats.
- This export layer is the main boundary between model inference and downstream 3D tools.
9. Interactive app layer
- The Gradio UI lives in src/depth_anything_3/app/.
- It is split into HTML/CSS, UI components, event handlers, file handlers, inference hooks, and visualization utilities.
- The app is a thin orchestration layer over the same backend/model pipeline used by the CLI.
10. CLI and operational modes
- cli.py exposes the command-line entry points documented in docs/CLI.md.
- The CLI supports at least three operational patterns visible in the README:
- backend-only caching
- auto processing over image sets
- video processing with export and feature visualization
- The same inference core is reused across these modes, which keeps behavior consistent between UI and batch workflows.
11. Benchmark subsystem
- src/depth_anything_3/bench/ is a separate evaluation pipeline with dataset adapters, evaluator logic, registries, and metric reporting.
- Dataset support includes DTU, ETH3D, HiRoom, ScanNet++, and 7Scenes.
- This part of the repo is for standardized evaluation of pose estimation, reconstruction, and rendering quality rather than inference serving.
12. Streaming extension
- da3_streaming/ extends DA3 for ultra-long sequence inference.
- It contains:
- streaming configs
- a fast loop-closure solver in C++ (fastloop/solve.cpp) with a Python wrapper
- loop detection, alignment, refinement, and Sim3 utilities
- The architecture here is a memory-efficient sliding-window pipeline with geometric correction across windows.
13. External dependencies that matter architecturally
- The repo relies on PyTorch for inference, Gradio for the UI, and Hugging Face model hosting for pretrained weights.
- gsplat is an optional dependency for Gaussian-head functionality.
- Triton appears in the alignment utilities, suggesting GPU-accelerated alternatives for some geometric kernels.
14. Practical architecture summary
- At a high level, the system is: input ingestion → config-driven model inference → geometric post-processing → export/visualization/evaluation.
- The major boundaries are between:
- frontend surfaces (CLI / Gradio / streaming)
- the reusable inference backend
- the model implementation
- export and benchmark consumers
- the streaming-specific long-sequence correction pipeline
Main entry points: da3.py DA3 net · api.py DA3 API · inference_service.py Inference svc · da3_streaming.py Streaming
Components 21
Surfaces
CLI, app, and reusable backend entry points.
- CLI · Command-line workflows for backend reuse, batch processing, and video inference.
src/depth_anything_3/cli.py - Gradio app · Interactive web UI built on the shared inference services.
src/depth_anything_3/app/gradio_app.py - Backend · Keeps model execution resident for repeated jobs and app reuse.
src/depth_anything_3/services/backend.py
Core inference
Shared runtime for loading models, processing inputs, and running inference.
- Inference svc · Coordinates reusable inference flows across UI and CLI surfaces.
src/depth_anything_3/services/inference_service.py - Input handlers · Handles images, folders, videos, and reference-view selection.
src/depth_anything_3/services/input_handlers.py - Input processor · Normalizes and batches inputs before model execution.
src/depth_anything_3/utils/io/input_processor.py - DA3 API · Main runtime façade over loading, normalization, inference, and packaging.
src/depth_anything_3/api.py
Model stack
Config-driven network assembly and task-specific heads.
- DA3 net · Neural network implementation behind the public API.
src/depth_anything_3/model/da3.py - Config registry · Loads YAML configs and instantiates objects dynamically.
src/depth_anything_3/cfg.py - DINOv2 · Transformer backbone used as the shared feature extractor.
src/depth_anything_3/model/dinov2/dinov2.py - Decoders · DPT-style decoder family for depth, dual, metric, and nested variants.
src/depth_anything_3/model/dpt.py - Camera heads · Camera encoding and decoding adapters for pose-related outputs.
src/depth_anything_3/model/cam_enc.py - GS adapter · Optional Gaussian-splatting path and renderer integration.
src/depth_anything_3/model/gs_adapter.py
Post-process
Geometry, output packaging, and export to downstream formats.
- Output processor · Packages processed frames, depth, confidence, and camera outputs.
src/depth_anything_3/utils/io/output_processor.py - Geometry · Shared ray, pose, and scene geometry helpers.
src/depth_anything_3/utils/geometry.py - Export · Exports inference results to visualization and 3D tool formats.
src/depth_anything_3/utils/export
Evaluation
Dataset adapters and benchmark execution.
- Bench runner · Coordinates standardized evaluation and metric reporting.
src/depth_anything_3/bench/evaluator.py - Datasets · Adapters for DTU, ETH3D, HiRoom, ScanNet++, and 7Scenes.
src/depth_anything_3/bench/datasets
Streaming
Long-sequence sliding-window inference and loop refinement.
- Streaming · Sliding-window inference for long videos and sequence-level correction.
da3_streaming/da3_streaming.py - Loop utils · Loop detection, alignment, Sim3 utilities, and refinement logic.
da3_streaming/loop_utils - Fast loop · C++ solver and Python wrapper for fast loop-closure optimization.
da3_streaming/fastloop/solve.cpp
Connections 20
- CLI → Inference svc: uses · CLI routes batch and video workflows through the shared service layer.
- Gradio app → Inference svc: uses · UI events call the same inference orchestration as the CLI.
- Backend → DA3 API: hosts · Resident backend keeps the model façade loaded for repeated jobs.
- Inference svc → Input handlers: ingests · Selects and prepares images, folders, videos, and reference views.
- Input handlers → Input processor: prepares · Turns user inputs into normalized batches and runtime-ready tensors.
- Inference svc → DA3 API: runs · Executes inference through the central façade.
- DA3 API → Config registry: loads · Resolves model variants and runtime objects from config.
- Config registry → DA3 net: constructs · Builds the underlying network from config-defined components.
- DA3 net → DINOv2: extracts features · Uses DINOv2 as the shared transformer encoder.
- DA3 net → Decoders: decodes · Applies DPT-style and variant-specific heads.
- DA3 net → Camera heads: predicts · Optionally attaches camera pose and camera-encoding branches.
- DA3 net → GS adapter: supports · Routes optional Gaussian-splatting outputs through adapter logic.
- DA3 API → Output processor: packages · Formats model results into frames, maps, confidence, and cameras.
- Output processor → Geometry: uses · Relies on geometry helpers for pose alignment and scene math.
- Output processor → Export: exports · Sends results to GLB, NPZ, COLMAP, depth, feature, and GS outputs.
- Bench runner → Datasets: evaluates · Runs standardized benchmarks over dataset-specific adapters.
- Streaming → DA3 API: reuses · Streaming inference reuses the same model façade where possible.
- Streaming → Loop utils: refines · Applies loop detection, alignment, and Sim3 correction across windows.
- Loop utils → Fast loop: accelerates · Uses the native solver for fast loop-closure optimization.
- Streaming → Input processor: windows · Processes long videos in sliding windows to bound memory usage.