# xpixelgroup/diffbir architecture

> Architecture diagram of the GitHub repository xpixelgroup/diffbir, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xpixelgroup/diffbir
- Repository: https://github.com/xpixelgroup/diffbir

## Overview

##### Project Type and Purpose
- DiffBIR is an open-source, full-pipeline blind image restoration toolkit that leverages a two-stage training setup (SwinIR degradation removal followed by a diffusion-based IRControlNet) and supports multiple restoration tasks (super-resolution, denoising, face restoration). It also provides a Gradio‐based UI, CLI, and modular captioners (LLaVA, RAM).

##### File Structure Analysis
- Top‐level:
- assets/: logos, documentation, visual examples
- configs/: YAML/JSON configs for inference and training (stage1 & stage2)
- diffbir/: core library
– dataset/: data loaders & degradation pipelines
– model/: ControlNet, UNet, VAE, SwinIR, SCUNet, BSRNet, CLDM, plus OpenCLIP
– sampler/: various diffusion samplers (DDIM, DPM, EDM, k‐diffusion…)
– inference/: loop orchestrators for each task
– pipeline.py: ties together dataset, model, sampler for inference
– utils/: captioning, common routines, face‐specific utilities
- llava/: multimodal captioner integration, evaluation scripts, serving modules
- ram/: alternative captioner (text‐only), data utilities, configs
- inference.py: CLI entry for batch inference
- run_gradio.py: launches Gradio UI
- train_stage1.py / train_stage2.py: two‐stage training scripts
- scripts/: converter utilities

- Components:
A. Data Sources
1. Training datasets (file lists → configs/train → train_stage1.py / train_stage2.py)
2. Inference inputs (inputs/demo or arbitrary image folders)
B. Training Pipeline
- Stage 1 (SwinIR): configs/train/train_stage1.yaml → train_stage1.py → SwinIR model checkpoint
- Stage 2 (IRControlNet): configs/train/train_stage2*.yaml → train_stage2.py → IRControlNet checkpoint
C. Core Library (diffbir)
1. dataset: batching, augmentations, degradation
2. model: instantiates SwinIR, UNet, ControlNet, VAE, CLDM, etc.
3. sampler: sampling strategies for diffusion
4. inference loops: BSR, BFR, BID, custom
5. pipeline.py: orchestrator that loads pre-trained weights + sampler + dataset → outputs restored images
D. Captioners (optional condition)
- llava/: multimodal encoder + language model interface
- ram/: tag2text, BERT, Swin backbone for lightweight captions
E. Interfaces
1. CLI: inference.py → calls pipeline.py → outputs results/
2. Gradio UI: run_gradio.py → uses llava.serve.gradio_web_server → invokes pipeline under the hood
F. External Dependencies & Services
- Pre-trained SD v2.1 checkpoint (weights/)
- Off-the-shelf models (BSRNet, SCUNet, SwinIR-Face) via pretrained_models.py
- mixture-of-diffusers, Tiled-VAE for patch-based sampling
- Hugging Face & BaiduNetdisk to host checkpoints

- Relationships & Data Flow
1. Training data → Stage1 trainer → SwinIR.ckpt
2. Training data + SwinIR.ckpt + SD v2.1 → Stage2 trainer → IRControlNet.ckpt
3. Inference input → (optional) Captioner → prompts/conditions
4. Inference input + prompts + SwinIR (stage1 removal) → VAE encode → CLDM diffusion guided by IRControlNet → VAE decode → restored output
5. CLI or Gradio UI drives (3)–(4).

##### Architectural Patterns & Design Principles
- Separation of Concerns: dataset ↔ model ↔ sampler ↔ inference orchestrator
- Two-Stage Training: modular swap of stage1 & stage2 checkpoints
- Plugin-style Captioning: llava and ram live as interchangeable modules
- Config-Driven: all hyperparameters live under configs/, enabling reproducible experiments
- Extensible Samplers: sampler implementations decoupled from core inference loops

##### Detailed Coverage Checklist
- [ ] Data ingestion: training file lists, inference folder
- [ ] Stage1 & Stage2 training modules + config
- [ ] Core pipeline: dataset → model → sampler → output
- [ ] Captioners & prompt conditioning path
- [ ] CLI vs. Gradio UI invocation paths
- [ ] External model & checkpoint repositories
- [ ] Tiled sampling extension points

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        TrainingData["Training Datasets"]:::external
        InferenceInput["Inference Input Folder"]:::external
    end

    %% Training Pipeline
    subgraph "Training Pipeline" 
        direction TB
        T1["train_stage1.py"]:::training
        ConfigTrain["configs/train/"]:::training
        SwinIRCkpt["SwinIR.ckpt"]:::training
        T2["train_stage2.py"]:::training
        Stage2Config["configs/train/"]:::training
        IRControlCkpt["IRControlNet.ckpt"]:::training
    end

    %% Core Library
    subgraph "Core Library (diffbir)" 
        direction TB
        Dataset["diffbir/dataset/"]:::core
        Model["diffbir/model/"]:::core
        Sampler["diffbir/sampler/"]:::core
        InferenceLoops["diffbir/inference/"]:::core
        Pipeline["diffbir/pipeline.py"]:::core
        Utils["diffbir/utils/"]:::core
    end

    %% Captioners
    subgraph "Captioners (optional)"
        direction TB
        LLava["llava/"]:::captioner
        LLavaServe["gradio_web_server.py"]:::captioner
        RAM["ram/"]:::captioner
    end

    %% Interfaces
    subgraph "Interfaces"
        direction TB
        CLI["inference.py"]:::interface
        Gradio["run_gradio.py"]:::interface
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        SD["SD v2.1 Checkpoint"]:::external
        HF["Hugging Face & BaiduNetdisk"]:::external
        OffModels["pretrained_models.py"]:::external
    end

    %% Training Flow
    TrainingData -->|configs/train| ConfigTrain
    ConfigTrain --> T1
    T1 --> SwinIRCkpt
    TrainingData -->|configs/train, SwinIR.ckpt, SD| T2
    Stage2Config --> T2
    T2 --> IRControlCkpt

    %% Inference Flow
    InferenceInput -->|image files| LLavaServe
    InferenceInput -->|image files| RAM
    LLavaServe -->|prompts| Pipeline
    RAM -->|tags| Pipeline
    InferenceInput -->|image| Pipeline

    %% Pipeline Internals
    Pipeline --> Dataset
    Pipeline --> Model
    Pipeline --> Sampler
    Pipeline --> Utils
    Pipeline --> InferenceLoops
    Model -->|uses SwinIR removal| Dataset
    Sampler -->|diffusion samples| Model
    InferenceLoops -->|orchestrates| Model
    Pipeline -->|restored images| Output["Restored Outputs"]:::external

    %% Interfaces to Pipeline
    CLI -->|calls| Pipeline
    Gradio -->|calls| Pipeline

    %% External to Training/Inference
    SD --> T2
    HF --> OffModels
    OffModels --> Model

    %% Click Events
    click ConfigTrain "https://github.com/xpixelgroup/diffbir/tree/main/configs/train/"
    click T1 "https://github.com/xpixelgroup/diffbir/blob/main/train_stage1.py"
    click SwinIRCkpt "https://github.com/xpixelgroup/diffbir/blob/main/train_stage1.py"
    click Stage2Config "https://github.com/xpixelgroup/diffbir/tree/main/configs/train/"
    click T2 "https://github.com/xpixelgroup/diffbir/blob/main/train_stage2.py"
    click IRControlCkpt "https://github.com/xpixelgroup/diffbir/blob/main/train_stage2.py"
    click Dataset "https://github.com/xpixelgroup/diffbir/tree/main/diffbir/dataset/"
    click Model "https://github.com/xpixelgroup/diffbir/tree/main/diffbir/model/"
    click Sampler "https://github.com/xpixelgroup/diffbir/tree/main/diffbir/sampler/"
    click InferenceLoops "https://github.com/xpixelgroup/diffbir/tree/main/diffbir/inference/"
    click Pipeline "https://github.com/xpixelgroup/diffbir/blob/main/diffbir/pipeline.py"
    click Utils "https://github.com/xpixelgroup/diffbir/tree/main/diffbir/utils/"
    click CLI "https://github.com/xpixelgroup/diffbir/blob/main/inference.py"
    click Gradio "https://github.com/xpixelgroup/diffbir/blob/main/run_gradio.py"
    click LLava "https://github.com/xpixelgroup/diffbir/tree/main/llava/"
    click LLavaServe "https://github.com/xpixelgroup/diffbir/blob/main/llava/serve/gradio_web_server.py"
    click RAM "https://github.com/xpixelgroup/diffbir/tree/main/ram/"
    click Output "https://github.com/xpixelgroup/diffbir/tree/main/assets/"

    %% Styles
    classDef training fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef core fill:#D8BFD8,stroke:#000,stroke-width:1px
    classDef interface fill:#FFA500,stroke:#000,stroke-width:1px
    classDef captioner fill:#FFE4B5,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
```
