How liheyoung/depth-anything works
Project Type and Purpose
• Depth Anything is a research-grade, full-stack machine-learning library and toolset for monocular depth estimation.
• It provides:
• Core relative-depth inference models (vision-transformer based)
• Fine-tuned metric-depth pipelines
• A depth-conditioned ControlNet module
• A semantic-segmentation downstream module
• CLI tools (run.py, run_video.py), a local Gradio demo (app.py), and integrations (HuggingFace Hub, ONNX, TensorRT).
File-Structure Analysis
• Top level:
• app.py: Gradio front end
• run.py / run_video.py: CLI inference scripts
• depth_anything/: core model implementation (encoders, transforms)
• metric_depth/: metric-depth training & evaluation pipelines
• controlnet/: improved depth-based ControlNet configs & code
• semseg/: semantic-segmentation configs & code
• torchhub/: packaging of Dinov2 for torch.hub
• assets/: sample inputs, outputs, paper, gallery
• Config files:
• JSON for ControlNet, semseg .py configs, environment.yml
• Build / deps:
• requirements.txt
Key README Insights
• Core “DepthAnything” class loads pre-trained ViT models via HF Hub or local weights
• Preprocessing pipeline uses depth_anything.util.transform
• Multiple downstream/extension points: metric_depth, controlnet, semseg
• External services: HuggingFace Spaces (demo), HF Hub (models), ONNX/TensorRT forks
Main Components
a. Input Layer
• Image Directory / Single Image / Text File / Video
• Asset Examples (assets/examples, assets/examples_video)
b. Preprocessing
• depth_anything.util.transform (Resize, Normalize, PrepareForNet)
• Configurable via CLI flags (--grayscale, --pred-only)
c. Core Model
• DepthAnything (from depth_anything.dpt)
• Three scales (Small/Base/Large) – loaded from HF Hub or locally
d. Postprocessing & Visualization
• Depth map coloring, side-by-side image+depth display
• Video frame stitching in run_video.py
• Gradio UI in app.py
e. Extension Modules
• Metric-Depth: fine-tuning & eval scripts (metric_depth/)
• ControlNet: retrained, config JSON, integration points (controlnet/)
• Semantic Segmentation: Mask2Former configs (semseg/)
f. External Services
• HuggingFace Hub (model checkpoints, Spaces demo)
• ONNX / TensorRT forks
• GitHub integrations (ControlNet WebUI, ComfyUI)
Relationships and Data Flows
• Inputs → Preprocessing → Core Model → Postprocessing → Outputs
• CLI tools and Gradio app consume the same preprocessing + model pipeline
• Extension modules tap into the core encoder for downstream tasks
• Config files steer ControlNet and semseg pipelines
• External services host or consume models (HF Hub, ONNX/TensorRT)
Architectural Patterns & Principles
• Modular ML Pipeline: clear separation between data I/O, transforms, model, and visualization
• Plugin-style extensions: metric_depth, controlnet, semseg each wrap the core encoder
• Config-driven: JSON and Python config files allow easy swapping of datasets/models
• CLI + GUI Dual Interface
Must-Include Elements
• Clear component names: “Preprocessing”, “DepthAnything Model”, “MetricDepth Trainer”, “ControlNet Plugin”, “SemSeg Plugin”
• Entry points: run.py, run_video.py, app.py