# zsyoaoa/resshift architecture

> Architecture diagram of the GitHub repository zsyoaoa/resshift, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zsyoaoa/resshift
- Repository: https://github.com/zsyoaoa/resshift

## Overview

##### Project Type and Purpose
- This is a Python-based full-stack machine-learning library and command-line application for *diffusion-based image super-resolution and restoration* (ResShift).
- Core purpose: train and run an *efficient diffusion model* that shifts residuals between low-res/ high-res images, dramatically reducing sampling steps while preserving image quality.
- Key workflows:
- Training: prepare data → train model via `main.py`/ `trainer.py` → log checkpoints.
- Inference/Serving: CLI tools (`inference_resshift.py`, `predict.py`), web demo (`app.py`) → input image(s) → output restored image(s).

##### File-Structure Analysis
Top-level:
- `main.py`, `trainer.py` – entry points for training.
- `inference_resshift.py`, `predict.py`, `app.py` – entry points for inference/serving.
- `configs/` – YAML configs for various tasks.
- `requirements.txt`, `environment.yml` – Python dependencies.
Core modules:
- basicsr/ – adaptation of BasicSR’s data pipelines, utilities (`data/`, `utils/`).
- datapipe/ – custom dataset definitions and augmentation/degradation modules.
- ldm/ – latent diffusion model implementation (VQ-VAE autoencoder, diffusion modules, attention, upscaling).
- models/ – helper classes: UNet, GaussianDiffusion, transformer blocks, loss definitions.
- utils/ – common utilities (file I/O, logging, option parsing).
Scripts & tools:
- `sampler.py`, `predict.py` – sampling utilities.
- `scripts/` – dataset preparation and metric computation.

a. Main Components
- Data Ingestion & Preprocessing
– *Modules*: basicsr/data, datapipe/
– *Files*: `data_sampler.py`, `transforms.py`, `aug.py`, `datasets.py`
- Model Core
– *Autoencoder*: ldm/models/autoencoder.py (VQ-VAE)
– *Diffusion Engine*: models/gaussian_diffusion.py, models/unet.py, ldm/modules/diffusionmodules/
– *Residual Shifting Pipeline*: orchestrated in `main.py`, `models/script_util.py`
- Training Orchestration
– *Entry Point*: `main.py`, `trainer.py`
– *Configuration*: YAML under `configs/`
– *Distributed Training*: torchrun + PyTorch Distributed Data Parallel
– *Logging & Checkpointing*: basicsr/utils/logger.py
- Inference & Serving
– *Scripts*: `inference_resshift.py`, `predict.py`
– *Web Demo*: `app.py` + Gradio interface
- Utilities & External Services
– *Utilities*: basicsr/utils, utils/ (common, network, options)
– *External Libraries*: PyTorch 2.x, xformers, BasicSR, Real-ESRGAN (for degradation), VQGAN weights
– *Storage*: local filesystem for images, checkpoints; optional cloud bucket for large datasets

b. Relationships & Data Flow
1. **Data Pipeline** → *DataLoader* → batches of (LQ, GT) images →
2. **Model** ← *training loop* ← receives data + config settings →
3. **Diffusion Process** inside Model → residual shifting steps → generate SR output →
4. **Loss Module** compares output vs. GT → gradients → backprop → weights update →
5. **Checkpoint & Logger** save metrics and model states.
6. **Inference Path**: Input image → `inference_resshift.py` loads checkpoint + config → runs diffusion sampler (15→4 steps) → outputs SR image → saved to disk or served via Gradio.

c. Architectural Patterns & Principles
- *Modularity*: clear separation between data, model, trainer, inference, and utilities.
- *Config-Driven*: every experiment controlled by YAML files, decoupling code from hyperparameters.
- *Pipeline Pattern*: sequential ML pipeline (ingest → preprocess → model → postprocess).
- *Reusability*: BasicSR and LDM upstream modules are wrapped/adapted, not rewritten.
- *Scalability*: supports single-machine multi-GPU distributed training.

d. Key Technologies & Frameworks
- PyTorch 2.x (+ xformers for efficient attention)
- BasicSR (data utilities)
- Latent Diffusion Model (LDM) components
- Real-ESRGAN (degradation synthesis)
- VQGAN (pretrained autoencoder weights)
- YAML config parsing (OmegaConf via basicsr/utils/options)
- Gradio (web demo)

##### Essential Elements
- Clear component labels matching directory names (e.g., “ldm/modules/diffusionmodules”).
- Callouts for external dependencies (PyTorch, xformers, VQGAN weights).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        RawImages["Raw Images"]:::data
        DataSampler["data_sampler.py"]:::data
        Transforms["transforms.py"]:::data
        Datasets["datasets.py"]:::data
        Augmentation["aug.py"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        VQAutoencoder["autoencoder.py (VQ-VAE)"]:::model
        DiffusionEngine["Diffusion Engine"]:::model
        ResidualShift["Residual Shift Pipeline"]:::model
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        MainTrain["main.py"]:::orchestration
        Trainer["trainer.py"]:::orchestration
        ConfigStore["YAML Configs"]:::util
        DistTrain["Distributed Dispatcher"]:::util
        Logger["logger.py"]:::util
        Checkpointer["Checkpoint Store"]:::storage
    end

    %% Inference Layer
    subgraph "Inference Layer"
        direction TB
        InferenceCLI["inference_resshift.py"]:::orchestration
        PredictScript["predict.py"]:::orchestration
        SamplerUtil["sampler.py"]:::orchestration
        WebDemo["app.py (Gradio)"]:::orchestration
    end

    %% Utilities & External
    subgraph "Utilities & External"
        direction TB
        OptionParsing["options.py / util_opts.py / util_common.py / util_image.py / util_net.py"]:::util
        DistUtil["dist_util.py"]:::util
        LossFuncs["losses.py"]:::model
        Degradation["BSRGAN Degradation"]:::model
        Quantize["quantize modules"]:::model
        Attention["attention.py"]:::model
        ExternalLibs["PyTorch, xformers, BasicSR, VQGAN weights"]:::external
        FileSystem["Local Filesystem"]:::storage
    end

    %% Data Flow Arrows
    RawImages -->|loads| DataSampler
    DataSampler --> Transforms
    Transforms --> Datasets
    Datasets --> Augmentation
    Augmentation -->|batch of images| VQAutoencoder
    VQAutoencoder -->|latent tensor| DiffusionEngine
    DiffusionEngine -->|residual signal| ResidualShift
    ResidualShift -->|SR output| LossFuncs

    %% Training Flow
    ConfigStore -->|config read| MainTrain
    MainTrain -->|orchestrates| Trainer
    Trainer -->|calls| VQAutoencoder
    Trainer -->|calls| DiffusionEngine
    Trainer -->|uses| DistTrain
    Trainer -->|logs metrics| Logger
    Trainer -->|saves model| Checkpointer
    LossFuncs -->|gradients| Trainer

    %% Inference Flow
    ConfigStore -->|config read| InferenceCLI
    InferenceCLI -->|loads checkpoint| Checkpointer
    InferenceCLI -->|runs sampler| SamplerUtil
    SamplerUtil -->|15→4 steps| DiffusionEngine
    SamplerUtil -->|SR output| FileSystem
    WebDemo -->|serves| DiffusionEngine
    WebDemo -->|loads checkpoint| Checkpointer

    %% Dependencies & Utilities
    OptionParsing -->|parses| MainTrain
    OptionParsing -->|parses| InferenceCLI
    DistUtil -->|enables| DistTrain
    Degradation -->|used by| DataSampler
    Quantize -->|used by| VQAutoencoder
    Attention -->|used by| DiffusionEngine
    ExternalLibs -->|provides| VQAutoencoder
    ExternalLibs -->|provides| DiffusionEngine

    %% Storage
    FileSystem -->|stores| RawImages
    FileSystem -->|stores| Checkpointer

    %% Click Events
    click DataSampler "https://github.com/zsyoaoa/resshift/blob/journal/basicsr/data/data_sampler.py"
    click Transforms "https://github.com/zsyoaoa/resshift/blob/journal/basicsr/data/transforms.py"
    click Datasets "https://github.com/zsyoaoa/resshift/blob/journal/datapipe/datasets.py"
    click Augmentation "https://github.com/zsyoaoa/resshift/blob/journal/datapipe/aug.py"
    click VQAutoencoder "https://github.com/zsyoaoa/resshift/blob/journal/ldm/models/autoencoder.py"
    click DiffusionEngine "https://github.com/zsyoaoa/resshift/blob/journal/models/gaussian_diffusion.py"
    click DiffusionEngine "https://github.com/zsyoaoa/resshift/blob/journal/models/unet.py"
    click DiffusionEngine "https://github.com/zsyoaoa/resshift/tree/journal/ldm/modules/diffusionmodules/"
    click ResidualShift "https://github.com/zsyoaoa/resshift/blob/journal/main.py"
    click ResidualShift "https://github.com/zsyoaoa/resshift/blob/journal/models/script_util.py"
    click MainTrain "https://github.com/zsyoaoa/resshift/blob/journal/main.py"
    click Trainer "https://github.com/zsyoaoa/resshift/blob/journal/trainer.py"
    click ConfigStore "https://github.com/zsyoaoa/resshift/blob/journal/configs/realsr_realesrgan256_x2.yaml"
    click Logger "https://github.com/zsyoaoa/resshift/blob/journal/basicsr/utils/logger.py"
    click InferenceCLI "https://github.com/zsyoaoa/resshift/blob/journal/inference_resshift.py"
    click PredictScript "https://github.com/zsyoaoa/resshift/blob/journal/predict.py"
    click SamplerUtil "https://github.com/zsyoaoa/resshift/blob/journal/sampler.py"
    click WebDemo "https://github.com/zsyoaoa/resshift/blob/journal/app.py"
    click OptionParsing "https://github.com/zsyoaoa/resshift/blob/journal/basicsr/utils/options.py"
    click OptionParsing "https://github.com/zsyoaoa/resshift/blob/journal/utils/util_opts.py"
    click OptionParsing "https://github.com/zsyoaoa/resshift/blob/journal/utils/util_common.py"
    click OptionParsing "https://github.com/zsyoaoa/resshift/blob/journal/utils/util_image.py"
    click OptionParsing "https://github.com/zsyoaoa/resshift/blob/journal/utils/util_net.py"
    click DistUtil "https://github.com/zsyoaoa/resshift/blob/journal/basicsr/utils/dist_util.py"
    click LossFuncs "https://github.com/zsyoaoa/resshift/blob/journal/models/losses.py"
    click Degradation "https://github.com/zsyoaoa/resshift/blob/journal/datapipe/degradation_bsrgan/bsrgan.py"
    click Degradation "https://github.com/zsyoaoa/resshift/blob/journal/datapipe/degradation_bsrgan/bsrgan_light.py"
    click Degradation "https://github.com/zsyoaoa/resshift/blob/journal/datapipe/degradation_bsrgan/utils_image.py"
    click Quantize "https://github.com/zsyoaoa/resshift/blob/journal/ldm/modules/quantize.py"
    click Quantize "https://github.com/zsyoaoa/resshift/blob/journal/ldm/modules/vqvae/quantize.py"
    click Attention "https://github.com/zsyoaoa/resshift/blob/journal/ldm/modules/attention.py"

    %% Styles
    classDef data fill:#add8e6,stroke:#000,stroke-width:1px
    classDef model fill:#90ee90,stroke:#000,stroke-width:1px
    classDef orchestration fill:#ffa500,stroke:#000,stroke-width:1px
    classDef util fill:#d3d3d3,stroke:#000,stroke-width:1px
    classDef storage fill:#f0e68c,stroke:#000,stroke-width:1px
    classDef external fill:#d3d3d3,stroke-dasharray: 5 5,stroke:#444
    class RawImages,FileSystem,Checkpointer storage
    class ExternalLibs external
```
