How blackjoke76/omnisr works
Project Type & Purpose
• This is a research codebase implementing a deep learning pipeline for shadow removal under direct/indirect lighting in RGB-D images.
• It includes data‐preprocessing, a PyTorch model with external feature extractors, training/inference scripts, and evaluation tools.
Main Components
A. Data Sources (Inputs)
• Shadow datasets (ISTD, ISTD+, SRD, WSRD+, INS, Real Photo)
• Pretrained weights for Depth-Anything-V2 and DINOv2
B. Preprocessing Pipeline
1. Depth Generation
• Depth-Anything-V2 (“Depth-Anything-V2_run.py”) → Depth maps
2. Normal Map Calculation
• calculate_normal.py → Normal maps from depth
C. Dataset Abstraction
• dataset.py & utils/dataset_utils.py → PyTorch Dataset + DataLoader:
– Loads RGB shadow image, shadow mask, depth, normal, ground truth
D. Core Model
• model.py → OmniSR network:
– Accepts RGB-D and semantic features
– Integrates DINO-V2 features (dinov2 folder)
– Computes direct/indirect shadow removal features
E. Training Infrastructure
1. Options & Configuration
• options.py → CLI args and hyperparameters (paths, win_size, train_ps)
2. Scheduler
• warmup_scheduler/scheduler.py & run.py → learning-rate warmup
3. Loss Functions
• losses.py → custom shadow‐removal losses
4. Training Scripts
• train.sh / train_DDP.py / test_DDP.py → Distributed training/inference
F. Evaluation
• evaluation/measure_shadow.m → MATLAB script for quantitative metrics
G. Utilities
• utils/*.py → helper functions (image_utils, model_utils, loader, etc.)
Relationships & Data Flow
1. External Data & Weights (side inputs) → Preprocessing modules
– Depth-Anything-V2 & calculate_normal.py produce depth/normal maps
2. Preprocessed Data → dataset.py / DataLoader
3. Dataloader → Core Model
4. Core Model ↔ Dinov2 feature extractor (external library)
5. Model Outputs + Ground Truth → losses.py → loss values
6. Losses → Scheduler → optimizer steps in train_DDP.py
7. Trained weights → test_DDP.py → inference outputs
8. Outputs → evaluation/measure_shadow.m → PSNR/SSIM results
Architectural Patterns & Design Principles
• Modular separation:
– Preprocessing vs. Model vs. Training vs. Evaluation
• Use of external feature extractor (Dinov2) as a pluggable component
• Distributed data‐parallel training (torch.distributed)
• Pipeline orchestration via shell scripts (train.sh/test.sh)