How mingcv/dsit works
Project Type and Purpose
• DSIT is a research-oriented deep-learning pipeline for Single Image Reflection Separation, based on an Interactive Dual-Stream Transformer.
• It provides data preparation, model architectures, training, evaluation, and testing scripts—no frontend/UI; it’s purely a Python-based ML framework.
File Structure
• Top-level scripts:
• train_sirs_setting1.py / train_sirs_setting2.py – entry points for training under two configurations.
• eval_sirs.py – evaluation entry point.
• test_sirs.py – testing/inference entry point.
• Core modules:
• data/ – dataset classes (sirs_dataset.py, image_folder.py), text files listing images, and transform definitions.
• models/ – model definitions:
– arch/ holds specific architectures (dsit.py, swin_det.py).
– base_model.py, networks.py, vgg.py, plus concrete wrappers like dsit_model_sirs_lrm.py.
– losses.py – loss functions.
• engine.py – training/evaluation loop implementation.
• options.py – command-line argument parsing.
• tools/ – utilities for checkpoint saving (saver.py), miscellaneous utilities (mutils.py).
• util/ – logging, HTML report generation (html.py, index.py), scheduler factories, visualization helpers.
• Config & dependencies:
• requirements.txt
• assets/ for logos and figures.
README
• Confirms environment (Python 3.9 + PyTorch + dependencies).
• Summarizes datasets (VOC, real-world paired sets, SIR²).
• Illustrates high-level network architecture (dual-stream transformer).
• Shows usage patterns (train/eval/test CLI commands).
• Points out pretrained backbone (Swin Transformer).
Main Components and Relationships
A. CLI Layer
– train_sirs_*.py, eval_sirs.py, test_sirs.py
– Invokes options.py → parses arguments → passes config to Engine.
B. Data Layer
– data/ module: Dataset classes, transforms
– Preprocessing pipeline: transforms → DataLoader
C. Model Layer
– models/arch/: DSIT core, Swin backbone
– base_model.py, networks.py, vgg.py
– dsit_model_sirs_lrm.py binds arch + loss + optional nature branch
D. Training Engine
– engine.py: orchestrates forward, backward, optimizer step
– interacts with Model, DataLoader, Loss, Scheduler
E. Loss & Metrics
– models/losses.py: reconstruction, perceptual (VGG) losses
F. Utilities
– tools/saver.py: checkpoint I/O
– util/visualizer.py, util/html.py, util/index.py: logging & report generation
– util/schedulers.py: LR schedulers
G. Output
– Saved model weights (weights/)
– HTML reports, tensorboard logs (if used)
– Evaluation metrics and visual comparisons
Architectural Patterns & Design Principles
• Modular separation:
• Data ↔ Model ↔ Engine with clear interfaces.
• CLI-driven configuration via options.py.
• Extensibility: easily swap architectures in models/arch/ or datasets in data/.
• Reuse of standard libraries (PyTorch, torchvision, timm for Swin).
a. Layout
– Left: Data Layer → DataLoader → Engine.
• calls Model (→ Forward)
• calls Loss → Backprop → Optimizer/Scheduler
– Right: Utilities & Output:
• checkpoint saver, visualizer, HTML report generator
• final weights & evaluation results
CLI ➔ Engine
Data ➔ Engine
Engine ➔ Model
Model & Ground-Truth ➔ Loss ➔ Engine
Engine ➔ Saver/Visualizer ➔ Output
• Blue for core components (Engine, Model, Data)
• Green for utilities (Saver, Visualizer)
• Orange for entry scripts (CLI)
• Datasets on disk
• Pretrained weights
d. Labels
– Annotate key classes/functions (e.g., DSITModel, SIRSDataSet, train_one_epoch).
Final Checklist
• [ ] Annotations for key technologies: PyTorch, timm (Swin), torchvision transforms
• [ ] Clear labeling of entry scripts and their parameters