# thomasmry/encdiff architecture

> Architecture diagram of the GitHub repository thomasmry/encdiff, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thomasmry/encdiff
- Repository: https://github.com/thomasmry/encdiff

## Overview

##### Main Project Components
- Data Management and Preprocessing:
- An external data dependency element that shows how external datasets are provided (as described in the README and dataset path structure).

- Configuration and Environment:
- “configs” directory with two main branches: one for the autoencoder (VQVAE/VAE) and one for latent diffusion.
- “environment.yaml” as the configuration for the conda environment ensuring reproducibility.

- Model Components:
- Autoencoder (VQVAE/VAE):
▪ Based on the configs in “configs/autoencoder,” the autoencoder is trained first to obtain latent codes.
- Latent Diffusion Module:
▪ Break down the latent diffusion submodules:
- Denoising U-Net (often shown in the diffusion pipelines),
- Diffusion sampler methods (e.g., ddpm, ddim, plms).
- Cross Attention Mechanism:
▪ Show cross attention modules bridging the output of the image encoder with the diffusion model.
▪ This part is critical as it provides the “inductive bias” where concept tokens (obtained from the encoder) are used as conditions in the diffusion process.

- Training Execution and Flow:
- “main.py” and “main_val.py” scripts as launch points for training (VQVAE/VAE training and subsequent EncDiff training).

- Evaluation and Metrics:
- Detail the evaluation frameworks as outlined in the “evaluation” folder:
▪ Evaluation metrics, abstract reasoning tests, and downstream tasks.
- That evaluation scripts (like “evaluate.py” in evaluation) validate the model’s performance.

##### Relationships and Interaction Flow
- Data → Image Encoder (Autoencoder training) → Latent Codes.
- Latent Codes + Concept Tokens are fed to the diffusion model.
- The cross attention module explicitly connects the encoder’s output (concept tokens) with the denoising U-Net in the latent diffusion step.
- Iterative training loop and checkpoint saving (logs’ folder reference from README).
- Data feedback loops from the evaluation component back to adjust model training parameters, if relevant.

##### Architectural Patterns and Technologies
- Modular design using configuration merging (OmegaConf) and the clear separation between VQVAE/VAE and latent diffusion.
- Directories (ldm/models, ldm/modules) as submodules representing defined functionalities (e.g., attention modules, diffusion modules, encoder implementations).

- Connectors from the autoencoder’s latent space output to the diffusion model’s conditioning input.
- On the side, include a block for “Evaluation & Metrics” that takes outputs from the diffusion pipeline and measures performance across various metrics.
- Finally, include all configuration management blocks (OmegaConf merging, config files from “configs”) as part of the orchestration layer.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External & Data"
        A["External Data"]:::external
        B["Data Preprocessing (data & ground truth)"]:::data
    end

    subgraph "Configuration & Environment"
        C1["Configs: Autoencoder"]:::config
        C2["Configs: Latent Diffusion"]:::config
        C3["Conda Environment"]:::config
    end

    subgraph "Autoencoder Training Phase"
        D["Image Encoder / VQVAE-VAE"]:::module
        E["Autoencoder Training Scripts"]:::training
        F["Latent Codes"]:::module
    end

    subgraph "Diffusion (EncDiff) Training Phase"
        G["Latent Diffusion Module"]:::module
        G1["Denoising U-Net"]:::module
        G2["Diffusion Sampler (ddpm/ddim/plms)"]:::module
        G3["Cross Attention Mechanism"]:::module
        H["Config Merging (OmegaConf)"]:::util
    end

    subgraph "Evaluation"
        I["Evaluation & Metrics"]:::eval
    end

    %% Data Flow
    A -->|"provides"| B
    B -->|"feeds"| D
    C3 -->|"sets env for"| D
    C1 -->|"configures"| D
    D -->|"produces"| F
    E -->|"triggers"| D
    F -->|"supplies latent codes to"| G
    C2 -->|"configures"| G

    %% Diffusion Submodules Flow
    G --> G1
    G --> G2
    D --"concept tokens"--> G3
    G3 --> G1

    %% Configuration Merging Flow
    H -->|"merges for encoder"| D
    H -->|"merges for diffusion"| G

    %% Evaluation Flow
    G1 -->|"outputs"| I
    G2 --> I
    G3 --> I

    %% Click Events
    click B "https://github.com/thomasmry/encdiff/tree/main/data/"
    click C1 "https://github.com/thomasmry/encdiff/tree/main/configs/autoencoder"
    click C2 "https://github.com/thomasmry/encdiff/tree/main/configs/latent-diffusion"
    click C3 "https://github.com/thomasmry/encdiff/blob/main/environment.yaml"
    click D "https://github.com/thomasmry/encdiff/blob/main/ldm/models/autoencoder.py"
    click E "https://github.com/thomasmry/encdiff/blob/main/main_val.py"
    click G1 "https://github.com/thomasmry/encdiff/blob/main/ldm/models/diffusion/ddpm_enc.py"
    click G2 "https://github.com/thomasmry/encdiff/blob/main/ldm/models/diffusion/ddim.py"
    click G3 "https://github.com/thomasmry/encdiff/blob/main/ldm/modules/attention.py"
    click H "https://github.com/thomasmry/encdiff/blob/main/ldm/util.py"
    click I "https://github.com/thomasmry/encdiff/blob/main/evaluation/evaluate.py"

    %% Styles
    classDef external fill:#a3d5ff,stroke:#333,stroke-width:2px;
    classDef data fill:#c7f7c7,stroke:#333,stroke-width:2px;
    classDef config fill:#f0e68c,stroke:#333,stroke-width:2px;
    classDef module fill:#ffa07a,stroke:#333,stroke-width:2px;
    classDef training fill:#ffdab9,stroke:#333,stroke-width:2px;
    classDef util fill:#d3d3f3,stroke:#333,stroke-width:2px;
    classDef eval fill:#90ee90,stroke:#333,stroke-width:2px;
```
