How ch-andrei/hdr-iqa-dom-adapt works
Project Type and Purpose
• The project is a deep learning research codebase focused on image quality assessment (IQA) for high dynamic range (HDR) displays.
• Its main purpose is to re-train preexisting neural network models (e.g., PieAPP, VTAMIQ) originally designed for standard dynamic range (SDR) data by incorporating domain adaptation techniques. This involves leveraging pre-trained networks and augmenting training with methods like DeepCORAL, plus exploring different normalization schemes (Pmax vs. 255).
• The repository is intended for experimentation and research, emphasizing training workflows, data preprocessing, and model modules.
Analysis of the File Structure
• Top-Level Directories:
• data: Manages datasets, script utilities, and data pre-/post-processing (including patch sampling and dataset splitting).
• modules: Contains various model components (e.g., PieAPP, VisionTransformer, DeepCORAL, RCAN, Attention modules, display simulation, and vtamiq) that implement different IQA methods or components of the overall architectures.
• training: Contains training scripts and configuration files for different training regimes (pre-training, domain adaptation, and multi-task training).
• utils: Provides ancillary functionalities such as image processing, logging, MATLAB interfacing, and miscellaneous helpers.
• Root-level scripts (e.g., run_pretrain_model.py and run_da.py): Serve as entry points for the training pipelines.
• The structure suggests a modular design where each key functionality (data loading, model definition, training procedures, and utility functions) is isolated, allowing clear separation of concerns.
Key Components
a. Data Layer:
• Datasets: Represented by the “data/datasets” directory and associated scripts (data/patch_sampling.py, data/utils.py, etc.). This module handles the ingestion and preprocessing (splitting, patch extraction) of both SDR and HDR images.
• Scripts for dataset preparation (e.g., upiq_split_compute.py) could be represented as sub-processes within this layer.
b. Model Modules:
• Core Models: Modules like PieAPP (modules/PieAPP), VTAMIQ (modules/vtamiq and modules/VisionTransformer), RCAN, and Attention (modules/Attention) show the variety of network architectures used.
• Auxiliary Modules: Modules for DeepCORAL (modules/DeepCORAL) and display simulation utilities show additional functionality designed to help with domain adaptation (i.e., aligning the deep feature activations between SDR and HDR domains).
c. Training Pipeline:
• Training scripts in “training” that orchestrate the training loops (train.py, train_da.py, train_multi.py) and configuration (train_config.py).
• Main entry points (run_pretrain_model.py and run_da.py) that kick off different phases of training.
• Loss composition (combining SDR loss, HDR loss, and the CORAL loss) is central to the training logic.
d. Utility/Support Functions:
• These utility components support both the data processing and training runtime environments.
Relationships and Interactions
• Data flows from the Data Layer (datasets and pre-processing scripts) into the Training Pipeline.
• The Training Pipeline then uses the Model Modules which encapsulate the neural network architectures.
• Specialized domain adaptation and loss modules (e.g., DeepCORAL from modules/DeepCORAL) are integrated into the training process.
• Utility functions provide cross-cutting support (e.g., for logging and image processing) and interact with both the data handling and training.
Important Architectural Patterns and Design Principles
• Modularity: Separation of data, models, training, and utilities to encourage reusability and easier maintenance.
• Pipeline Structure: A well-defined multi-phase training pipeline – starting with pre-training on SDR data and proceeding to domain adaptation training with HDR data.
• Layered Architecture: Distinct separation of core functionalities similar to an application with a “data layer”, “service layer” (Model modules and Loss modules) and “interface layer” (training scripts that serve as launchers).
• Dependency Inversion: Model components and utilities are designed to be plug-and-play, where training scripts call on model modules according to configuration.