# zhaozixiang1228/mmif-ddfm architecture

> Architecture diagram of the GitHub repository zhaozixiang1228/mmif-ddfm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhaozixiang1228/mmif-ddfm
- Repository: https://github.com/zhaozixiang1228/mmif-ddfm

## Overview

##### Project Type and Purpose
- The project is a research-oriented image fusion tool using a denoising diffusion probabilistic model (DDPM) framework. Its purpose is to fuse images—such as infrared-visible or medical images—by treating fusion as a conditional generation problem. It leverages pre-trained unconditional models and integrates an EM (Expectation-Maximization) based inference strategy.

##### Main Components
a. Input Data and Configuration:
- “input/”: This directory contains the actual image data (with separate subdirectories for different modalities like infrared “ir” and visible “vi”).
- “configs/”: Contains YAML configuration files (e.g., diffusion_config.yaml, model_config_imagenet.yaml) that set parameters like timestep_respacing and model settings.

b. Diffusion Engine (Backbone) – Guided Diffusion Module:
- “guided_diffusion/”: Houses the core diffusion process implementations.
- Files like gaussian_diffusion.py encapsulate the DDPM sampling logic.
- “unet.py” provides the neural network (Unet) used in the denoising steps.
- “EM_onestep.py” integrates the inference steps based on the EM algorithm.
- Other support files include “condition_methods.py”, “posterior_mean_variance.py” and “fp16_util.py” for precision handling.
- It also includes an __init__.py for package initialization and organizing module references.

c. Inference Runner:
- “sample.py”: Acts as the entry point for inference. It loads the configuration, inputs, and the pre-trained diffusion checkpoint (downloaded externally as per the README) to generate fused images.

d. Utility Functions:
- “util/”: Contains helper modules such as “logger.py” for logging and “pytorch_colors.py” for visualization or debugging purposes.

e. Pre-trained Model and External Dependencies:
- While not in the file tree, the README states that a pre-trained checkpoint “256x256_diffusion_uncond.pt” (from OpenAI’s guided diffusion repo) must be downloaded and placed in a “models” folder.
- External libraries (e.g., PyTorch as noted in the requirements.txt) play a key role in the model’s functioning.

##### Relationships and Interactions
- The Inference Process (sample.py) is the driver that:
- Loads configurations and input image data.
- Pulls in the pre-trained guided diffusion model to perform fusion.
- Invokes functions from the ‘guided_diffusion’ submodules (e.g., the Unet for denoising and EM modules for latent variable estimation).
- The diffusion engine components interact via defined interfaces: for example, condition methods supply modality-specific data into the diffusion process, while posterior estimation functions refine the outcome.
- Utility modules support logging and additional processing throughout the pipeline.

##### Key Architectural Patterns and Design Principles
- Modular Design: The separation of concerns is evident—with pure diffusion logic in “guided_diffusion”, configuration management in “configs”, and utility functionalities in “util”.
- Pipeline-based Processing: The design follows a multi-step process (configure → input loading → diffusion processing → inference → output generation).
- Integration of Statistical Inference: The use of the EM algorithm (visible through EM_onestep.py) integrated into the diffusion process highlights a hierarchical Bayesian approach.
- Loose Coupling: Each module is designed to work independently (e.g., the Unet is a self-contained neural network module).

##### Putting It All Together
- Then, incorporate the “Utilities” as a side-support layer that interfaces with both the “Core Engine” and the “Execution Layer”.
- Finally, include the pre-trained checkpoint as an external resource connected to the “Diffusion Engine.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["sample.py"]:::execution
    subgraph "Data & Configuration"
        B["Input Data"]:::data
        C["Diffusion Config"]:::data
        D["Model Config"]:::data
    end
    subgraph "Diffusion Engine"
        E["Gaussian Diffusion"]:::diffusion
        F["Unet"]:::diffusion
        G["EM Module"]:::diffusion
        H["Condition Methods"]:::diffusion
        I["Posterior Estimation"]:::diffusion
        J["FP16 Util"]:::diffusion
        K["Package Initialization"]:::diffusion
    end
    subgraph "Utility Functions"
        L["Logger"]:::utility
        M["PyTorch Colors"]:::utility
    end
    O["Pre-trained Checkpoint"]:::external

    %% Connections from Inference (sample.py) to Data & Configuration
    A -->|"reads"| B
    A -->|"reads"| C
    A -->|"reads"| D

    %% Connection from Inference to Diffusion Engine
    A -->|"invokes"| E

    %% Data flowing into Diffusion Engine
    B -->|"supplies"| E
    C -->|"supplies"| E
    D -->|"supplies"| E
    H -->|"supplies"| E

    %% Internal interactions within Diffusion Engine
    E -->|"denoises"| F
    E -->|"applies"| G
    E -->|"refines"| I
    E -->|"supports"| J
    E -->|"initializes"| K

    %% Connection from Diffusion Engine to External Checkpoint
    E -->|"loads_checkpoint"| O

    %% Utility interactions
    A -->|"logging"| L
    A -->|"debug"| M

    %% Styles
    classDef execution fill:#F9E79F,stroke:#D68910,stroke-width:2px;
    classDef data fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef diffusion fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef utility fill:#D4EFDF,stroke:#27AE60,stroke-width:2px;
    classDef external fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;

    %% Click Events
    click A "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/sample.py"
    click B "https://github.com/zhaozixiang1228/mmif-ddfm/tree/main/input/"
    click C "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/configs/diffusion_config.yaml"
    click D "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/configs/model_config_imagenet.yaml"
    click E "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/gaussian_diffusion.py"
    click F "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/unet.py"
    click G "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/EM_onestep.py"
    click H "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/condition_methods.py"
    click I "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/posterior_mean_variance.py"
    click J "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/fp16_util.py"
    click K "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/guided_diffusion/__init__.py"
    click L "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/util/logger.py"
    click M "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/util/pytorch_colors.py"
    click O "https://github.com/zhaozixiang1228/mmif-ddfm/blob/main/models/256x256_diffusion_uncond.pt"
```
