# blackjoke76/omnisr architecture

> Architecture diagram of the GitHub repository blackjoke76/omnisr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blackjoke76/omnisr
- Repository: https://github.com/blackjoke76/omnisr

## Overview

##### Project Type & Purpose
- This is a research codebase implementing a deep learning pipeline for shadow removal under direct/indirect lighting in RGB-D images.
- It includes data‐preprocessing, a PyTorch model with external feature extractors, training/inference scripts, and evaluation tools.

##### Main Components
A. Data Sources (Inputs)
- Shadow datasets (ISTD, ISTD+, SRD, WSRD+, INS, Real Photo)
- Pretrained weights for Depth-Anything-V2 and DINOv2
B. Preprocessing Pipeline
1. Depth Generation
- Depth-Anything-V2 (“Depth-Anything-V2_run.py”) → Depth maps
2. Normal Map Calculation
- calculate_normal.py → Normal maps from depth
C. Dataset Abstraction
- dataset.py & utils/dataset_utils.py → PyTorch Dataset + DataLoader:
– Loads RGB shadow image, shadow mask, depth, normal, ground truth
D. Core Model
- model.py → OmniSR network:
– Accepts RGB-D and semantic features
– Integrates DINO-V2 features (dinov2 folder)
– Computes direct/indirect shadow removal features
E. Training Infrastructure
1. Options & Configuration
- options.py → CLI args and hyperparameters (paths, win_size, train_ps)
2. Scheduler
- warmup_scheduler/scheduler.py & run.py → learning-rate warmup
3. Loss Functions
- losses.py → custom shadow‐removal losses
4. Training Scripts
- train.sh / train_DDP.py / test_DDP.py → Distributed training/inference
F. Evaluation
- evaluation/measure_shadow.m → MATLAB script for quantitative metrics
G. Utilities
- utils/*.py → helper functions (image_utils, model_utils, loader, etc.)

##### Relationships & Data Flow
1. External Data & Weights (side inputs) → Preprocessing modules
– Depth-Anything-V2 & calculate_normal.py produce depth/normal maps
2. Preprocessed Data → dataset.py / DataLoader
3. Dataloader → Core Model
4. Core Model ↔ Dinov2 feature extractor (external library)
5. Model Outputs + Ground Truth → losses.py → loss values
6. Losses → Scheduler → optimizer steps in train_DDP.py
7. Trained weights → test_DDP.py → inference outputs
8. Outputs → evaluation/measure_shadow.m → PSNR/SSIM results

##### Architectural Patterns & Design Principles
- Modular separation:
– Preprocessing vs. Model vs. Training vs. Evaluation
- Use of external feature extractor (Dinov2) as a pluggable component
- Distributed data‐parallel training (torch.distributed)
- Pipeline orchestration via shell scripts (train.sh/test.sh)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        DS1[(ISTD, ISTD+, SRD, WSRD+, INS, Real Photo)]:::dataSource
        DS2[(Pretrained Weights)]:::dataSource
        DW1(["Depth-Anything-V2_run.py"]):::dataSource
        DW2[(Depth-Anything-V2)]:::dataSource
        DINO[(dinov2)]:::dataSource
    end

    %% Preprocessing
    subgraph "Preprocessing" 
        direction TB
        PG1["Depth Generation"]:::preproc
        PG2(["calculate_normal.py"]):::preproc
    end

    %% Dataset Abstraction
    subgraph "Dataset Abstraction"
        direction TB
        DSAB1(["dataset.py"]):::config
        DSAB2(["utils/dataset_utils.py"]):::config
    end

    %% Core Model
    subgraph "Core Model"
        direction TB
        CM1(["model.py"]):::coreModel
    end

    %% Training Infrastructure
    subgraph "Training Infrastructure"
        direction TB
        CFG(["options.py"]):::config
        SCH1(["warmup_scheduler/run.py"]):::config
        SCH2(["warmup_scheduler/scheduler.py"]):::config
        LS(["losses.py"]):::coreModel
        SCR1[/train.sh/]:::script
        TDDP(["train_DDP.py"]):::script
        SCR2[/test.sh/]:::script
        TeDDP(["test_DDP.py"]):::script
    end

    %% Evaluation
    subgraph "Evaluation"
        direction TB
        EV1[(measure_shadow.m)]:::evaluation
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        U1(["utils/antialias.py"]):::utilities
        U2(["utils/bundle_submissions.py"]):::utilities
        U3(["utils/dir_utils.py"]):::utilities
        U4(["utils/image_utils.py"]):::utilities
        U5(["utils/loader.py"]):::utilities
        U6(["utils/misc.py"]):::utilities
        U7(["utils/model_utils.py"]):::utilities
        U8(["utils/shadow_mask_evaluate.py"]):::utilities
    end

    %% Data Flow
    DS1 --> DW1
    DS2 --> DW1
    DW1 --> PG1
    PG1 --> PG2
    PG2 --> DSAB1
    DSAB1 --> CM1
    DSAB2 --> DSAB1
    DINO --> CM1
    CM1 --> LS
    LS --> SCH1
    SCH1 --> TDDP
    CFG -.-> TDDP
    TDDP --> SCR2
    SCR2 --> TeDDP
    TeDDP --> EV1

    %% Utilities Dependencies
    U1 -.-> CM1
    U2 -.-> SCR1
    U3 -.-> DSAB1
    U4 -.-> DSAB1
    U5 -.-> DSAB1
    U6 -.-> CM1
    U7 -.-> CM1
    U8 -.-> EV1

    %% Click Events
    click DW1 "https://github.com/blackjoke76/omnisr/blob/main/Depth-Anything-V2_run.py"
    click DW2 "https://github.com/blackjoke76/omnisr/tree/main/Depth-Anything-V2/"
    click DINO "https://github.com/blackjoke76/omnisr/tree/main/dinov2/"
    click PG2 "https://github.com/blackjoke76/omnisr/blob/main/calculate_normal.py"
    click DSAB1 "https://github.com/blackjoke76/omnisr/blob/main/dataset.py"
    click DSAB2 "https://github.com/blackjoke76/omnisr/blob/main/utils/dataset_utils.py"
    click CM1 "https://github.com/blackjoke76/omnisr/blob/main/model.py"
    click CFG "https://github.com/blackjoke76/omnisr/blob/main/options.py"
    click SCH1 "https://github.com/blackjoke76/omnisr/blob/main/warmup_scheduler/run.py"
    click SCH2 "https://github.com/blackjoke76/omnisr/blob/main/warmup_scheduler/scheduler.py"
    click LS "https://github.com/blackjoke76/omnisr/blob/main/losses.py"
    click SCR1 "https://github.com/blackjoke76/omnisr/blob/main/train.sh"
    click TDDP "https://github.com/blackjoke76/omnisr/blob/main/train_DDP.py"
    click SCR2 "https://github.com/blackjoke76/omnisr/blob/main/test.sh"
    click TeDDP "https://github.com/blackjoke76/omnisr/blob/main/test_DDP.py"
    click EV1 "https://github.com/blackjoke76/omnisr/blob/main/evaluation/measure_shadow.m"
    click U1 "https://github.com/blackjoke76/omnisr/blob/main/utils/antialias.py"
    click U2 "https://github.com/blackjoke76/omnisr/blob/main/utils/bundle_submissions.py"
    click U3 "https://github.com/blackjoke76/omnisr/blob/main/utils/dir_utils.py"
    click U4 "https://github.com/blackjoke76/omnisr/blob/main/utils/image_utils.py"
    click U5 "https://github.com/blackjoke76/omnisr/blob/main/utils/loader.py"
    click U6 "https://github.com/blackjoke76/omnisr/blob/main/utils/misc.py"
    click U7 "https://github.com/blackjoke76/omnisr/blob/main/utils/model_utils.py"
    click U8 "https://github.com/blackjoke76/omnisr/blob/main/utils/shadow_mask_evaluate.py"

    %% Styles
    classDef dataSource fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef preproc fill:#d4edda,stroke:#333,stroke-width:1px
    classDef coreModel fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef script fill:#e2b6ff,stroke:#333,stroke-width:1px
    classDef evaluation fill:#d1ecf1,stroke:#333,stroke-width:1px
    classDef config fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef utilities fill:#f0f0f0,stroke:#333,stroke-width:1px
```
