# medarc-ai/fmri-reconstruction-nsd architecture

> Architecture diagram of the GitHub repository medarc-ai/fmri-reconstruction-nsd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/medarc-ai/fmri-reconstruction-nsd
- Repository: https://github.com/medarc-ai/fmri-reconstruction-nsd

## Overview

##### Project Type and Purpose
- MindEye is an end-to-end machine-learning pipeline for reconstructing and retrieving images from fMRI brain activity.
- It provides:
- A high-level pipeline (“MindEye”) mapping fMRI → CLIP embeddings → diffusion prior → images
- A low-level pipeline mapping fMRI → Variational Autoencoder (Stable Diffusion VAE)
- Inference scripts for reconstruction, retrieval (including LAION-5B), and evaluation of image metrics

##### High-Level View from the File Structure
- Top-level: README, docs, LICENSE, scripts for environment setup (setup.sh, download.sh), HPC job script (accel.slurm)
- src/ directory contains:
- Jupyter notebooks & corresponding .py scripts for Training (Train_MindEye), Reconstructions, Retrievals, Reconstruction_Metrics
- train_autoencoder.py for the low-level pipeline
- Modules: configurator.py (CLI config parser), models.py (network definitions), convnext.py (VICRegL backbone), utils.py (general utilities)
- environment.yaml for conda environment, plus helper scripts

##### Key Insights from README
- Data sources: Natural Scenes Dataset (NSD), optional LAION-5B, pretrained checkpoints from Huggingface
- Workflow steps: environment setup → training MindEye → optional low-level autoencoder training → reconstruct/infer → evaluate metrics
- Model artifacts saved under train_logs/, intermediate .pt outputs in src/

a. Main Components & Their Responsibilities
1. Data Ingestion Layer
- NSD Data Store (local or downloaded)
- LAION-5B index (for retrieval)
- Huggingface checkpoint repo (for diffusion prior & autoencoder weights)
2. Environment & Orchestration
- Conda Environment (environment.yaml)
- Setup & Download Scripts (setup.sh, download.sh)
- HPC Scheduler (accel.slurm)
3. Model Training Services
- High-Level Trainer (Train_MindEye.py)
- Reads fMRI → CLIP embeddings via models.py + convnext.py
- Trains diffusion prior loss, optional projector, logs to Wandb
- Low-Level Trainer (train_autoencoder.py)
- Trains fMRI → VAE mapping using VICRegL ConvNext + SD autoencoder
4. Model Artifact Storage
- Local train_logs/ directory
- Optional Wandb Artifacts
5. Inference Services
- Reconstruction Service (Reconstructions.py)
- Loads trained model(s), NSD fMRI inputs, outputs reconstructed images (.pt)
- Retrieval Service (Retrievals.py)
- Loads CLIP-mapping checkpoints, queries LAION-5B, performs second-order selection
6. Evaluation Component
- Metrics Service (Reconstruction_Metrics.py)
- Compares reconstructed/retrieved images vs ground truth, computes low- & high-level metrics
7. Documentation & Notebooks
- Jupyter Notebooks for explorability and manual runs

b. Relationships and Data Flow
- After training: Model Artifacts → Inference Services
- Inference Services → Generated Outputs (.pt files)
- Outputs + Ground Truth → Metrics Service → Evaluation Reports

c. Architectural Patterns & Principles
- Modular CLI-driven pipelines (each script is a separate micro-pipeline)
- Clear separation of concerns: training vs inference vs evaluation
- Reusable code modules (configurator, models, utils) shared across pipelines
- Data-centric ML architecture: raw data → features → model → artifacts → inference → metrics

d. Key Technologies & Frameworks
- Python, PyTorch, Huggingface Diffusers (Versatile Diffusion, Stable Diffusion VAE)
- CLIP (OpenAI variants), VICRegL ConvNext
- Conda, Slurm (HPC), Wandb (optional)
- WebDataset loading for NSD, torch .pt serialization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Point
    Entry[". setup.sh / conda activate mindeye"]:::process --> SetupScript

    subgraph "Environment & Orchestration"
        SetupScript["setup.sh"]:::process
        CondaEnv["environment.yaml"]:::data
        HPC["accel.slurm"]:::process
    end

    SetupScript --> CondaEnv
    SetupScript --> HPC

    subgraph "Data Ingestion Layer"
        DownloadScript["download.sh"]:::process
        NSD["NSD Data Store"]:::data
        LAION["LAION-5B Index"]:::external
        HF["Huggingface Checkpoints"]:::external
    end

    CondaEnv --> DownloadScript
    DownloadScript --> NSD
    DownloadScript --> LAION
    DownloadScript --> HF

    subgraph "Shared Utility Modules"
        Config["configurator.py"]:::process
        Models["models.py"]:::process
        Convnext["convnext.py"]:::process
        Utils["utils.py"]:::process
    end

    subgraph "Training Phase"
        HLTrainer["Train_MindEye.py"]:::process
        HLNotebook["Train_MindEye.ipynb"]:::process
        LLTrainer1["train_autoencoder.py"]:::process
        LLTrainer2["train_autoencoder_bold.py"]:::process
    end

    NSD --> HLTrainer
    HF --> HLTrainer
    NSD --> LLTrainer1
    HF --> LLTrainer1
    Convnext --> HLTrainer
    Models --> HLTrainer
    Config --> HLTrainer
    Utils --> HLTrainer
    Convnext --> LLTrainer1
    Models --> LLTrainer1
    Config --> LLTrainer1
    Utils --> LLTrainer1
    HLNotebook --> HLTrainer

    subgraph "Model Artifact Storage"
        ModelArtifacts["train_logs/ (Model Artifacts)"]:::data
        Wandb["Wandb"]:::external
    end

    HLTrainer --> ModelArtifacts
    LLTrainer1 --> ModelArtifacts
    HLTrainer --> Wandb
    LLTrainer1 --> Wandb

    subgraph "Inference Services"
        ReconScript["Reconstructions.py"]:::process
        ReconNB["Reconstructions.ipynb"]:::process
        RetScript["Retrievals.py"]:::process
        RetNB["Retrievals.ipynb"]:::process
    end

    ModelArtifacts --> ReconScript
    ModelArtifacts --> RetScript
    NSD --> ReconScript
    HF --> RetScript
    Config --> ReconScript
    Utils --> ReconScript
    Config --> RetScript
    Utils --> RetScript
    ReconNB --> ReconScript
    RetNB --> RetScript

    subgraph "Generated Outputs"
        OutputFiles["Reconstructed Outputs (.pt)"]:::data
    end

    ReconScript --> OutputFiles
    RetScript --> OutputFiles

    subgraph "Evaluation Component"
        MetricsScript["Reconstruction_Metrics.py"]:::process
        MetricsNB["Reconstruction_Metrics.ipynb"]:::process
        Reports["Evaluation Reports (metrics.csv)"]:::data
    end

    OutputFiles --> MetricsScript
    NSD --> MetricsScript
    Config --> MetricsScript
    Utils --> MetricsScript
    MetricsNB --> MetricsScript
    MetricsScript --> Reports

    subgraph "Documentation & Diagrams"
        PipelineDiag["pipeline.png"]:::external
        Readme["README.md"]:::external
        License["LICENSE"]:::external
    end

    PipelineDiag -.-> Entry
    Readme -.-> Entry
    License -.-> Entry

    %% Click Events
    click DownloadScript "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/download.sh"
    click SetupScript "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/setup.sh"
    click CondaEnv "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/environment.yaml"
    click HPC "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/accel.slurm"
    click HLTrainer "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Train_MindEye.py"
    click HLNotebook "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Train_MindEye.ipynb"
    click LLTrainer1 "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/train_autoencoder.py"
    click LLTrainer2 "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/train_autoencoder_bold.py"
    click ReconScript "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Reconstructions.py"
    click ReconNB "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Reconstructions.ipynb"
    click RetScript "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Retrievals.py"
    click RetNB "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Retrievals.ipynb"
    click MetricsScript "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Reconstruction_Metrics.py"
    click MetricsNB "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/Reconstruction_Metrics.ipynb"
    click Config "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/configurator.py"
    click Models "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/models.py"
    click Convnext "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/convnext.py"
    click Utils "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/src/utils.py"
    click PipelineDiag "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/docs/pipeline.png"
    click Readme "https://github.com/medarc-ai/fmri-reconstruction-nsd/blob/main/README.md"
    click License "https://github.com/medarc-ai/fmri-reconstruction-nsd/tree/main/LICENSE"

    %% Styles
    classDef data fill:#ccf,stroke:#000,color:#000
    classDef process fill:#cfc,stroke:#000,color:#000
    classDef external fill:#fcf08b,stroke:#000,color:#000
```
