How xpixelgroup/diffbir works
Project Type and Purpose
• DiffBIR is an open-source, full-pipeline blind image restoration toolkit that leverages a two-stage training setup (SwinIR degradation removal followed by a diffusion-based IRControlNet) and supports multiple restoration tasks (super-resolution, denoising, face restoration). It also provides a Gradio‐based UI, CLI, and modular captioners (LLaVA, RAM).
File Structure Analysis
• Top‐level:
• assets/: logos, documentation, visual examples
• configs/: YAML/JSON configs for inference and training (stage1 & stage2)
• diffbir/: core library
– dataset/: data loaders & degradation pipelines
– model/: ControlNet, UNet, VAE, SwinIR, SCUNet, BSRNet, CLDM, plus OpenCLIP
– sampler/: various diffusion samplers (DDIM, DPM, EDM, k‐diffusion…)
– inference/: loop orchestrators for each task
– pipeline.py: ties together dataset, model, sampler for inference
– utils/: captioning, common routines, face‐specific utilities
• llava/: multimodal captioner integration, evaluation scripts, serving modules
• ram/: alternative captioner (text‐only), data utilities, configs
• inference.py: CLI entry for batch inference
• run_gradio.py: launches Gradio UI
• train_stage1.py / train_stage2.py: two‐stage training scripts
• scripts/: converter utilities
• Components:
A. Data Sources
1. Training datasets (file lists → configs/train → train_stage1.py / train_stage2.py)
2. Inference inputs (inputs/demo or arbitrary image folders)
B. Training Pipeline
• Stage 1 (SwinIR): configs/train/train_stage1.yaml → train_stage1.py → SwinIR model checkpoint
• Stage 2 (IRControlNet): configs/train/train_stage2*.yaml → train_stage2.py → IRControlNet checkpoint
C. Core Library (diffbir)
1. dataset: batching, augmentations, degradation
2. model: instantiates SwinIR, UNet, ControlNet, VAE, CLDM, etc.
3. sampler: sampling strategies for diffusion
4. inference loops: BSR, BFR, BID, custom
5. pipeline.py: orchestrator that loads pre-trained weights + sampler + dataset → outputs restored images
D. Captioners (optional condition)
• llava/: multimodal encoder + language model interface
• ram/: tag2text, BERT, Swin backbone for lightweight captions
E. Interfaces
1. CLI: inference.py → calls pipeline.py → outputs results/
2. Gradio UI: run_gradio.py → uses llava.serve.gradio_web_server → invokes pipeline under the hood
F. External Dependencies & Services
• Pre-trained SD v2.1 checkpoint (weights/)
• Off-the-shelf models (BSRNet, SCUNet, SwinIR-Face) via pretrained_models.py
• mixture-of-diffusers, Tiled-VAE for patch-based sampling
• Hugging Face & BaiduNetdisk to host checkpoints
• Relationships & Data Flow
1. Training data → Stage1 trainer → SwinIR.ckpt
2. Training data + SwinIR.ckpt + SD v2.1 → Stage2 trainer → IRControlNet.ckpt
3. Inference input → (optional) Captioner → prompts/conditions
4. Inference input + prompts + SwinIR (stage1 removal) → VAE encode → CLDM diffusion guided by IRControlNet → VAE decode → restored output
5. CLI or Gradio UI drives (3)–(4).
Architectural Patterns & Design Principles
• Separation of Concerns: dataset ↔ model ↔ sampler ↔ inference orchestrator
• Two-Stage Training: modular swap of stage1 & stage2 checkpoints
• Plugin-style Captioning: llava and ram live as interchangeable modules
• Config-Driven: all hyperparameters live under configs/, enabling reproducible experiments
• Extensible Samplers: sampler implementations decoupled from core inference loops
Detailed Coverage Checklist
• [ ] Data ingestion: training file lists, inference folder
• [ ] Stage1 & Stage2 training modules + config
• [ ] Core pipeline: dataset → model → sampler → output
• [ ] Captioners & prompt conditioning path
• [ ] CLI vs. Gradio UI invocation paths
• [ ] External model & checkpoint repositories
• [ ] Tiled sampling extension points