How bbaaii/dreamdiffusion works
Project type and overall purpose
• Project type: Research-oriented machine learning pipeline for “thoughts-to-image” generation
• Purpose:
– Pretrain an EEG encoder via masked modeling (sc_mbm)
– Fine-tune a Stable Diffusion latent model with the EEG encoder as a conditioning module (dc_ldm)
– Generate and evaluate images from raw EEG signals
File structure and README
• Top-level folders:
– assets: teaser and pipeline images (for documentation only)
– code: all ML modules and entry-point scripts
– pretrains & datasets (external, not in repo): store pretrained checkpoints and raw data
• Key subdirectories in code/:
– sc_mbm/: masked autoencoder modules for EEG, trainer, utils
– dc_ldm/: latent diffusion modules/models adapted from Stable Diffusion
– Top-level scripts: stageA1_eeg_pretrain.py, eeg_ldm.py, gen_eval_eeg.py
– Shared helpers: config.py, dataset.py, eval_metrics.py
Main components
1) Data Storage
– EEG raw signals (MOABB, .pth splits)
– Image dataset (ImageNet subset)
– Pretrained Stable Diffusion checkpoint (v1-5-pruned.ckpt)
2) Pre-training Pipeline (Stage A1)
– Entry script: stageA1_eeg_pretrain.py
– Modules: sc_mbm/mae_for_eeg.py, trainer.py, utils.py
– Output: pre-trained EEG encoder checkpoint → pretrains/eeg_pretrain/checkpoint.pth
3) Fine-tuning Pipeline (Stage B)
– Entry script: eeg_ldm.py
– Modules: dc_ldm/ldm_for_eeg.py, dc_ldm/models, dc_ldm/modules
– Conditioning: load EEG encoder + Stable Diffusion weights, optimize cross-attention heads
– Output: generation checkpoint → pretrains/generation/checkpoint_best.pth
4) Inference & Evaluation
– Entry script: gen_eval_eeg.py
– Loads generation checkpoint, dataset, eval_metrics.py
– Produces generated images + quantitative metrics
5) Configuration & Environment
– config.py: hyperparameters for all scripts
– env.yaml: conda environment specification
Relationships & data flow
• Data → Pre-training: dataset.py loads EEG splits → sc_mbm trainer pretrains encoder
• Pretrained Encoder → Fine-tuning: stageA1 checkpoint loaded by eeg_ldm script
• Stable Diffusion Model → Fine-tuning: external v1-5 checkpoint loaded into dc_ldm modules
• Fine-tuned Model → Inference: gen_eval_eeg loads generation checkpoint for sampling
• Evaluation: gen_eval_eeg calls eval_metrics to compute scores on generated outputs
Architectural patterns & design principles
• Pipeline pattern: three sequential stages (Pretrain → Fine-tune → Infer)
• Modular separation:
– sc_mbm for EEG representation learning
– dc_ldm for diffusion machinery
– Shared scripts/utilities for config, data loading, evaluation
• Dependency injection: config.py centrally controls hyperparams
• Plugin-style model modules: dc_ldm/models and modules mirror LDM codebase