# bbaaii/dreamdiffusion architecture

> Architecture diagram of the GitHub repository bbaaii/dreamdiffusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bbaaii/dreamdiffusion
- Repository: https://github.com/bbaaii/dreamdiffusion

## Overview

##### Project type and overall purpose
- Project type: Research-oriented machine learning pipeline for “thoughts-to-image” generation
- Purpose:
– Pretrain an EEG encoder via masked modeling (sc_mbm)
– Fine-tune a Stable Diffusion latent model with the EEG encoder as a conditioning module (dc_ldm)
– Generate and evaluate images from raw EEG signals

##### File structure and README
- Top-level folders:
– assets: teaser and pipeline images (for documentation only)
– code: all ML modules and entry-point scripts
– pretrains & datasets (external, not in repo): store pretrained checkpoints and raw data
- Key subdirectories in code/:
– sc_mbm/: masked autoencoder modules for EEG, trainer, utils
– dc_ldm/: latent diffusion modules/models adapted from Stable Diffusion
– Top-level scripts: stageA1_eeg_pretrain.py, eeg_ldm.py, gen_eval_eeg.py
– Shared helpers: config.py, dataset.py, eval_metrics.py

##### Main components
1) Data Storage
– EEG raw signals (MOABB, .pth splits)
– Image dataset (ImageNet subset)
– Pretrained Stable Diffusion checkpoint (v1-5-pruned.ckpt)
2) Pre-training Pipeline (Stage A1)
– Entry script: stageA1_eeg_pretrain.py
– Modules: sc_mbm/mae_for_eeg.py, trainer.py, utils.py
– Output: pre-trained EEG encoder checkpoint → pretrains/eeg_pretrain/checkpoint.pth
3) Fine-tuning Pipeline (Stage B)
– Entry script: eeg_ldm.py
– Modules: dc_ldm/ldm_for_eeg.py, dc_ldm/models, dc_ldm/modules
– Conditioning: load EEG encoder + Stable Diffusion weights, optimize cross-attention heads
– Output: generation checkpoint → pretrains/generation/checkpoint_best.pth
4) Inference & Evaluation
– Entry script: gen_eval_eeg.py
– Loads generation checkpoint, dataset, eval_metrics.py
– Produces generated images + quantitative metrics
5) Configuration & Environment
– config.py: hyperparameters for all scripts
– env.yaml: conda environment specification

##### Relationships & data flow
- Data → Pre-training: dataset.py loads EEG splits → sc_mbm trainer pretrains encoder
- Pretrained Encoder → Fine-tuning: stageA1 checkpoint loaded by eeg_ldm script
- Stable Diffusion Model → Fine-tuning: external v1-5 checkpoint loaded into dc_ldm modules
- Fine-tuned Model → Inference: gen_eval_eeg loads generation checkpoint for sampling
- Evaluation: gen_eval_eeg calls eval_metrics to compute scores on generated outputs

##### Architectural patterns & design principles
- Pipeline pattern: three sequential stages (Pretrain → Fine-tune → Infer)
- Modular separation:
– sc_mbm for EEG representation learning
– dc_ldm for diffusion machinery
– Shared scripts/utilities for config, data loading, evaluation
- Dependency injection: config.py centrally controls hyperparams
- Plugin-style model modules: dc_ldm/models and modules mirror LDM codebase

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Storage
    subgraph "Data Storage"
        DS1["EEG Raw Signals"]:::data
        DS2["Image Dataset"]:::data
        DS3["Stable Diffusion Checkpoint"]:::data
        Doc["Documentation Assets"]:::data
    end

    %% Pre-training Pipeline
    subgraph "Pre-training Pipeline\n(Stage A1)"
        PS["stageA1_eeg_pretrain.py"]:::preScript
        MAE["mae_for_eeg.py"]:::preModule
        TR["trainer.py"]:::preModule
        UT["utils.py"]:::preModule
        OUT1["EEG Encoder Checkpoint"]:::data
    end

    %% Fine-tuning Pipeline
    subgraph "Fine-tuning Pipeline\n(Stage B)"
        FS["eeg_ldm.py"]:::fineScript
        Cond["LDM Conditioning Module\n(ldm_for_eeg.py)"]:::fineModule
        LDM["Latent Diffusion Models\n(autoencoder.py)"]:::fineModule
        Sched["Diffusion Schedulers & Samplers"]:::fineModule
        Att["Cross-Attention & Blocks"]:::fineModule
        Wrap["Diffusion Module Wrappers"]:::fineModule
        Enc["Encoder Plugins"]:::fineModule
        Loss["Loss Functions"]:::fineModule
        EMA["EMA & Distribution Utilities"]:::fineModule
        UtilsFT["Shared DC-LDM Utilities"]:::fineModule
        OUT2["Generation Checkpoint"]:::data
    end

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        IS["gen_eval_eeg.py"]:::infScript
        EM["eval_metrics.py"]:::infModule
        GI["Generated Images & Metrics"]:::inf
    end

    %% Configuration & Environment
    subgraph "Configuration & Environment"
        CFG["config.py"]:::util
        ENV["env.yaml"]:::util
    end

    %% Data flow arrows
    DS1 -->|"loads EEG splits"| PS
    PS -->|"orchestrates"| MAE
    PS -->|"orchestrates"| TR
    PS -->|"uses"| UT
    TR -->|"saves checkpoint"| OUT1

    OUT1 -->|"loads pretrained encoder"| FS
    DS3 -->|"loads SD weights"| Cond
    FS -->|"optimizes"| Cond
    Cond -->|"calls models"| LDM
    Cond -->|"calls schedulers"| Sched
    Cond -->|"calls attention"| Att
    Cond -->|"calls wrappers"| Wrap
    Cond -->|"calls enc plugins"| Enc
    Cond -->|"calls loss"| Loss
    Cond -->|"uses EMA & utils"| EMA
    Cond -->|"uses shared utils"| UtilsFT
    FS -->|"saves checkpoint"| OUT2

    OUT2 -->|"loads generation checkpoint"| IS
    IS -->|"calls eval metrics"| EM
    IS -->|"produces"| GI

    %% Click Events
    click Doc "https://github.com/bbaaii/dreamdiffusion/blob/main/assets/eeg_teaser.pdf"
    click PS "https://github.com/bbaaii/dreamdiffusion/blob/main/code/stageA1_eeg_pretrain.py"
    click MAE "https://github.com/bbaaii/dreamdiffusion/blob/main/code/sc_mbm/mae_for_eeg.py"
    click TR "https://github.com/bbaaii/dreamdiffusion/blob/main/code/sc_mbm/trainer.py"
    click UT "https://github.com/bbaaii/dreamdiffusion/blob/main/code/sc_mbm/utils.py"
    click FS "https://github.com/bbaaii/dreamdiffusion/blob/main/code/eeg_ldm.py"
    click Cond "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/ldm_for_eeg.py"
    click LDM "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/models/autoencoder.py"
    click Sched "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/models/diffusion/ddpm.py"
    click Att "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/modules/attention.py"
    click Wrap "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/modules/diffusionmodules/model.py"
    click Enc "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/modules/encoders/modules.py"
    click Loss "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/modules/losses/contperceptual.py"
    click EMA "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/modules/ema.py"
    click UtilsFT "https://github.com/bbaaii/dreamdiffusion/blob/main/code/dc_ldm/util.py"
    click IS "https://github.com/bbaaii/dreamdiffusion/blob/main/code/gen_eval_eeg.py"
    click EM "https://github.com/bbaaii/dreamdiffusion/blob/main/code/eval_metrics.py"
    click CFG "https://github.com/bbaaii/dreamdiffusion/blob/main/code/config.py"
    click ENV "https://github.com/bbaaii/dreamdiffusion/blob/main/env.yaml"

    %% Styles
    classDef data fill:#cfe2f3,stroke:#2f75b5,color:#0b5394
    classDef preScript fill:#d9ead3,stroke:#38761d,color:#274e13,stroke-dasharray: 5 5
    classDef preModule fill:#d9ead3,stroke:#38761d,color:#274e13
    classDef fineScript fill:#fce5cd,stroke:#b45f06,color:#783f04,stroke-dasharray: 5 5
    classDef fineModule fill:#fce5cd,stroke:#b45f06,color:#783f04
    classDef infScript fill:#d9d2e9,stroke:#674ea7,color:#351c75,stroke-dasharray: 5 5
    classDef infModule fill:#d9d2e9,stroke:#674ea7,color:#351c75
    classDef inf fill:#d9d2e9,stroke:#674ea7,color:#351c75
    classDef util fill:#f2f2f2,stroke:#7f7f7f,color:#3f3f3f,stroke-dasharray: 2 2
```
