# sinzlab/energy-guided-diffusion architecture

> Architecture diagram of the GitHub repository sinzlab/energy-guided-diffusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sinzlab/energy-guided-diffusion
- Repository: https://github.com/sinzlab/energy-guided-diffusion

## Overview

##### Project type and purpose
- This repository is an open‑source research tool implementing “Energy Guided Diffusion” (EGG) on top of OpenAI’s guided‑diffusion framework. Its goal is to generate neurally exciting images by combining a pre‑trained diffusion model with a user‑defined energy function (e.g. neural response predictor).

##### File structure
Top‑level directories and files:
- Dockerfile, docker‑compose.yml, .env‑placeholder: define a reproducible environment.
- README.md, docs/: user documentation and experiment instructions.
- assets/: static images (e.g. teaser).
- download_requirements.sh: fetches large external packages.
- setup.py: installs this project as a Python package.

Core library “egg/”:
- egg/diffusion.py: the EGG wrapper around guided diffusion.
- egg/guided_diffusion/: a vendored copy of OpenAI’s guided‑diffusion code (data sampling, model definitions, training utilities).
- egg/models/: model definitions split into “data_driven” and “task_driven” cores (e.g. grayscale model implementations).
- egg/optimizers/: plug‑in optimizers (e.g. gradient_ascent).
- egg/palettes.py: visualization helpers.

Auxiliary code:
- lib/: pip‑installable local packages (nnvision, mei) providing neural‐vision utilities.
- scripts/: CLI scripts to run full experiments (e.g. meis_egg.py, menis.py, reconstruct_gd.py).
- notebooks/: Jupyter notebooks for analysis and plotting results.

##### README for additional insights
- Describes usage: how to instantiate `EGG(diffusion_artefact, num_steps)`, supply an `energy_fn` (neural predictor), and call `.sample()`.
- Explains how to build the Docker image (`docker compose build base`) and run experiments via `docker compose run --rm python scripts/...`.
- Emphasizes reproducibility: containerization + pip editable installs.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface
    subgraph "User Interface"
        CLI_Jupyter["CLI / Jupyter Frontend"]:::extension
    end

    %% Infrastructure Layer
    subgraph "Infrastructure" 
        Dockerfile["Dockerfile"]:::infra
        DockerCompose["docker-compose.yml"]:::infra
        Env[".env‑placeholder"]:::infra
    end

    %% External Dependencies
    subgraph "External Dependencies"
        Pretrained["Pre‑trained Diffusion Artefact"]:::external
        Attention["Attention‑Readout Weights"]:::external
        Lib["lib/__init__.py"]:::external
    end

    %% Core Library
    subgraph "EGG Core Library"
        Sampler["EGG Sampler\n(egg/diffusion.py)"]:::core
        subgraph "Guided Diffusion Engine"
            GD_init["__init__.py"]:::core
            GD_dist["dist_util.py"]:::core
            GD_fp16["fp16_util.py"]:::core
            GD_gauss["gaussian_diffusion.py"]:::core
            GD_data["image_datasets.py"]:::core
            GD_log["logger.py"]:::core
            GD_loss["losses.py"]:::core
            GD_nn["nn.py"]:::core
            GD_resample["resample.py"]:::core
            GD_respace["respace.py"]:::core
            GD_script["script_util.py"]:::core
            GD_train["train_util.py"]:::core
            GD_unet["unet.py"]:::core
        end
        subgraph "Model Zoo"
            subgraph "Data‑Driven"
                MD_gray["gray.py"]:::core
            end
            subgraph "Task‑Driven"
                MT_gray["gray.py"]:::core
            end
        end
        subgraph "Optimizers"
            Optimizer["gradient_ascent.py"]:::core
        end
        subgraph "Visualization"
            Palettes["palettes.py"]:::core
        end
    end

    %% Entry-Point Scripts
    subgraph "Scripts" 
        S1["meis_egg.py"]:::script
        S2["meis_ga.py"]:::script
        S3["menis.py"]:::script
        S4["get_full_receptive_field.py"]:::script
        S5["reconstruct_egg.py"]:::script
        S6["reconstruct_egg_real.py"]:::script
        S7["reconstruct_gd.py"]:::script
        S8["reconstruct_gd_real.py"]:::script
    end

    %% Notebooks
    subgraph "Analysis & Notebooks"
        N1["ablation_train.ipynb"]:::script
        N2["cka.ipynb"]:::script
        N3["fid_menis.ipynb"]:::script
        N4["human_study.ipynb"]:::script
        N5["meis.ipynb"]:::script
        N6["menis.ipynb"]:::script
        N7["reconstructions.ipynb"]:::script
    end

    %% Outputs
    subgraph "Outputs"
        Images["Generated Images, Logs, Metrics"]:::extension
    end

    %% Connections
    CLI_Jupyter -->|"run experiment"| DockerCompose
    DockerCompose -->|"containerized env"| Sampler
    CLI_Jupyter -->|"run experiment"| S1
    S1 -->|"invoke .sample()"| Sampler
    Sampler -->|"forward diffusion"| GD_init
    Sampler -->|"load diffusion weights"| Pretrained
    Sampler -->|"load neural predictor"| Attention
    Sampler -->|"use energy_fn"| MD_gray
    Sampler -->|"use energy_fn"| MT_gray
    Sampler -->|"gradient updates"| Optimizer
    Sampler -->|"visualize"| Palettes
    Lib -->|"provides utilities"| Sampler
    N1 -->|"analyze results"| Images
    N2 -->|"analyze results"| Images
    N3 -->|"analyze results"| Images
    N4 -->|"analyze results"| Images
    N5 -->|"analyze results"| Images
    N6 -->|"analyze results"| Images
    N7 -->|"analyze results"| Images

    %% Click Events - Infrastructure
    click Dockerfile "https://github.com/sinzlab/energy-guided-diffusion/tree/main/Dockerfile"
    click DockerCompose "https://github.com/sinzlab/energy-guided-diffusion/blob/main/docker-compose.yml"
    click Env "https://github.com/sinzlab/energy-guided-diffusion/blob/main/.env‑placeholder"

    %% Click Events - Core Library
    click Sampler "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/diffusion.py"
    click GD_init "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/__init__.py"
    click GD_dist "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/dist_util.py"
    click GD_fp16 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/fp16_util.py"
    click GD_gauss "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/gaussian_diffusion.py"
    click GD_data "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/image_datasets.py"
    click GD_log "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/logger.py"
    click GD_loss "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/losses.py"
    click GD_nn "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/nn.py"
    click GD_resample "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/resample.py"
    click GD_respace "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/respace.py"
    click GD_script "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/script_util.py"
    click GD_train "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/train_util.py"
    click GD_unet "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/guided_diffusion/unet.py"
    click MD_gray "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/models/data_driven/gray.py"
    click MT_gray "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/models/task_driven/gray.py"
    click Optimizer "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/optimizers/gradient_ascent.py"
    click Palettes "https://github.com/sinzlab/energy-guided-diffusion/blob/main/egg/palettes.py"

    %% Click Events - External
    click Lib "https://github.com/sinzlab/energy-guided-diffusion/blob/main/lib/__init__.py"

    %% Click Events - Scripts
    click S1 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/meis_egg.py"
    click S2 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/meis_ga.py"
    click S3 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/menis.py"
    click S4 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/get_full_receptive_field.py"
    click S5 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/reconstruct_egg.py"
    click S6 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/reconstruct_egg_real.py"
    click S7 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/reconstruct_gd.py"
    click S8 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/scripts/reconstruct_gd_real.py"

    %% Click Events - Notebooks
    click N1 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/ablation_train.ipynb"
    click N2 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/cka.ipynb"
    click N3 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/fid_menis.ipynb"
    click N4 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/human_study.ipynb"
    click N5 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/meis.ipynb"
    click N6 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/menis.ipynb"
    click N7 "https://github.com/sinzlab/energy-guided-diffusion/blob/main/notebooks/reconstructions.ipynb"

    %% Styles
    classDef core fill:#8ecae6,stroke:#023047,color:#000
    classDef extension fill:#a8dadc,stroke:#023047,color:#000
    classDef external fill:#ffb703,stroke:#fb8500,color:#000
    classDef script fill:#b5179e,stroke:#7209b7,color:#fff
    classDef infra fill:#adb5bd,stroke:#6c757d,color:#000
```
