How medarc-ai/fmri-reconstruction-nsd works
Project Type and Purpose
• MindEye is an end-to-end machine-learning pipeline for reconstructing and retrieving images from fMRI brain activity.
• It provides:
• A high-level pipeline (“MindEye”) mapping fMRI → CLIP embeddings → diffusion prior → images
• A low-level pipeline mapping fMRI → Variational Autoencoder (Stable Diffusion VAE)
• Inference scripts for reconstruction, retrieval (including LAION-5B), and evaluation of image metrics
High-Level View from the File Structure
• Top-level: README, docs, LICENSE, scripts for environment setup (setup.sh, download.sh), HPC job script (accel.slurm)
• src/ directory contains:
• Jupyter notebooks & corresponding .py scripts for Training (Train_MindEye), Reconstructions, Retrievals, Reconstruction_Metrics
• train_autoencoder.py for the low-level pipeline
• Modules: configurator.py (CLI config parser), models.py (network definitions), convnext.py (VICRegL backbone), utils.py (general utilities)
• environment.yaml for conda environment, plus helper scripts
Key Insights from README
• Data sources: Natural Scenes Dataset (NSD), optional LAION-5B, pretrained checkpoints from Huggingface
• Workflow steps: environment setup → training MindEye → optional low-level autoencoder training → reconstruct/infer → evaluate metrics
• Model artifacts saved under train_logs/, intermediate .pt outputs in src/
a. Main Components & Their Responsibilities
1. Data Ingestion Layer
• NSD Data Store (local or downloaded)
• LAION-5B index (for retrieval)
• Huggingface checkpoint repo (for diffusion prior & autoencoder weights)
2. Environment & Orchestration
• Conda Environment (environment.yaml)
• Setup & Download Scripts (setup.sh, download.sh)
• HPC Scheduler (accel.slurm)
3. Model Training Services
• High-Level Trainer (Train_MindEye.py)
• Reads fMRI → CLIP embeddings via models.py + convnext.py
• Trains diffusion prior loss, optional projector, logs to Wandb
• Low-Level Trainer (train_autoencoder.py)
• Trains fMRI → VAE mapping using VICRegL ConvNext + SD autoencoder
4. Model Artifact Storage
• Local train_logs/ directory
• Optional Wandb Artifacts
5. Inference Services
• Reconstruction Service (Reconstructions.py)
• Loads trained model(s), NSD fMRI inputs, outputs reconstructed images (.pt)
• Retrieval Service (Retrievals.py)
• Loads CLIP-mapping checkpoints, queries LAION-5B, performs second-order selection
6. Evaluation Component
• Metrics Service (Reconstruction_Metrics.py)
• Compares reconstructed/retrieved images vs ground truth, computes low- & high-level metrics
7. Documentation & Notebooks
• Jupyter Notebooks for explorability and manual runs
b. Relationships and Data Flow
• After training: Model Artifacts → Inference Services
• Inference Services → Generated Outputs (.pt files)
• Outputs + Ground Truth → Metrics Service → Evaluation Reports
c. Architectural Patterns & Principles
• Modular CLI-driven pipelines (each script is a separate micro-pipeline)
• Clear separation of concerns: training vs inference vs evaluation
• Reusable code modules (configurator, models, utils) shared across pipelines
• Data-centric ML architecture: raw data → features → model → artifacts → inference → metrics
d. Key Technologies & Frameworks
• Python, PyTorch, Huggingface Diffusers (Versatile Diffusion, Stable Diffusion VAE)
• CLIP (OpenAI variants), VICRegL ConvNext
• Conda, Slurm (HPC), Wandb (optional)
• WebDataset loading for NSD, torch .pt serialization