How ppwangyc/wave works
Main Components
• Environment & Setup: The project uses scripts (in the script/ folder) for setting up the conda environment, downloading datasets and model checkpoints via git-lfs. The env.yaml and setup.py scripts play a key role.
• Data Handling:
• Dataset: In src/dataset (e.g., bold5000.py), data loaders are implemented to process the BOLD5000 fMRI dataset.
• Batcher: In src/batcher, specialized batchers (including one based on BERT) prepare data for training.
• Model Components:
• Embedder: (src/embedder) for mapping fMRI data to a representation space using models like CSM and possibly unembedding.
• Model: In src/model, multiple modules (clip, diffusion, fid_wrapper, fvl) are available, with an emphasis on a diffusion model and a couple of contrastive learning models.
• Decoder: (src/decoder) which includes a GPT-based decoder, perhaps used for generating or interpreting latent representations.
• Trainer: (src/trainer) encapsulates training routines such as prior_trainer and general trainer that manage the training loop.
• Reconstruction & Evaluation:
• The src/reconstruct.py script aggregates model outputs to generate reconstructed visuals and computes relevant metrics (see src/util/metrics.py and src/util/utils.py).
Relationships & Data Flow
• The initial data (BOLD5000 fMRI images) is loaded via the dataset components; batcher modules process them into mini-batches.
• In the contrastive branch, the embedder maps data into embedding space, which is then used by the contrastive model that is trained via train scripts (src/train_bold5000_contrastive.py).
• The diffusion branch (in src/train_bold5000_decode.py) takes the outputs (or embeddings) from the contrastive model as a prior and performs the decoding/reconstruction.
• Reconstruction operations in src/reconstruct.py then integrate these components, use external metrics (from src/util), and finally output visual reconstructions.
• Scripts in the script/ directory act as the orchestrators by calling appropriate training and reconstruction commands, managing parameters like subject names and mask usages.
Architectural Patterns and Design Principles
• Modularity: The project is divided into clear modules for dataset, batcher, embedder, model, decoder, trainer, and utility functions.
• Pipeline-based design: There is a clear sequential flow—from loading and preprocessing the data, to training and embedding via contrastive learning, followed by decoding through diffusion.
• Separation of concerns: The training, reconstruction, and environment setup codes are clearly separated into different directories and scripts.
• Extensibility: Use of “make.py” modules in several subdirectories (batcher, decoder, embedder, model) implies a factory design pattern to simplify component instantiation.
• Split into two pipelines:
• Pipeline A (Contrastive Learning):
o Embedder: processes fMRI data into latent representations (src/embedder).
o Contrastive Model Training: using src/train_bold5000_contrastive.py, and components in src/model.
• Pipeline B (Diffusion Decoding):
o Diffusion Model: processes the learned representations to decode the fMRI data (src/model/diffusion.py).
o Decoder: further refines the output (src/decoder) to generate the final visual reconstruction.
• Both pipelines converge at the Reconstruction module (src/reconstruct.py) that evaluates and refines outputs.
• An "Experiment/Logging" block to represent integration with logging tools (such as wandb) and metrics (src/util/metrics.py).