How medarc-ai/cortexmae works
1. Repository purpose and top-level boundary
CortexMAE is a Python package that provides pretrained masked autoencoder models for fMRI, plus the training and inference code needed to reproduce them. The repo has two main package roots: cortex_mae for the model system itself, and brainmarks.models for integration into the external Brainmarks probing framework.
2. Core model subsystem
The heart of the repo lives in src/cortex_mae/models_mae.py, modules.py, masking.py, and nisc.py. This layer implements the MAE-style architecture: patch/token construction, masking strategy, transformer encoder/decoder components, and the Cortex-specific input representations. These files form the model boundary that everything else feeds into.
3. Input representation and preprocessing
The model supports three input spaces called out in the README: parcel, flat, and volume. The preprocessing path is handled by data.py, transforms.py, and utils.py, which prepare fMRI time series into model-ready tensors. The flat-map representation also depends on the packaged resource resources/flat_pca_n8_224_560.npz, indicating a learned or fixed projection used in the flat input pipeline.
4. Training pipeline
main_pretrain.py is the executable entry point for pretraining. It reads settings from config/default_pretrain.yaml, builds the data pipeline, instantiates the model, and runs masked autoencoder training. The architecture here is “config-driven training”: defaults are externalized in YAML, and command-line overrides can switch input space, learning rate, or model variant without changing code.
5. Inference and embedding generation
inference.py exposes the runtime path for turning an fMRI file into embeddings, which is what the README’s CortexMAE.from_pretrained(...).run_embedding(...) flow uses. This is the primary inference boundary: raw or preprocessed fMRI enters, and patch-level embeddings or related outputs leave. visualization.py likely supports interpreting those embeddings or reconstructions.
6. Pretrained model distribution
The repo is designed to consume pretrained weights rather than only train from scratch. The README references HuggingFace-hosted checkpoints, and the package exposes model loading helpers via the public API in src/cortex_mae/__init__.py. This means the runtime architecture includes an external model registry/service boundary, not just local checkpoints.
7. Data sources and storage boundaries
Two distinct data infrastructures appear in the architecture: HCP-YA pretraining data stored as webdataset shards on R2, and benchmark datasets stored as HuggingFace Arrow format on the MedARC R2 bucket via Brainmarks. The repo therefore straddles both streamed training data and separately managed downstream evaluation datasets, with AWS/R2 credentials required for access.
8. Brainmarks integration for downstream evaluation
src/brainmarks/models/cortex_mae_wrapper.py adapts CortexMAE encoders to the Brainmarks API. This is a deliberate boundary layer: CortexMAE owns model construction and embeddings, while Brainmarks owns probe evaluation, task selection, and benchmarking orchestration. The README’s brainmarks.main_probe invocation reflects this split.
9. Public package surface
src/cortex_mae/__init__.py defines the user-facing API, centered on CortexMAE and helpers like resolve_file. This is the main stable entry point for consumers who want to load pretrained checkpoints, resolve remote files, and run embeddings without touching the internal training stack.
10. Tests as subsystem coverage
The test suite is organized around the main architectural units: test_models.py, test_modules.py, test_transforms.py, and test_inference.py. There is also a wrapper test in brainmarks/models/test_cortex_mae_wrapper.py, which helps confirm the integration boundary remains compatible with Brainmarks.
11. Packaging and release workflow
pyproject.toml, MANIFEST.in, and the GitHub Actions workflow in .github/workflows/publish.yml show that this is a distributable Python package with automated publishing. The repo also includes uv.lock and README installation instructions for uv, so reproducible environment management is part of the delivery path.
12. Documentation and usage examples
notebooks/quickstart.ipynb is the primary interactive demo and mirrors the README’s inference flow. It functions as a reference integration layer rather than core logic, showing how the package is intended to be used end-to-end with pretrained weights and remote fMRI files.
13. Overall data flow
End-to-end, the architecture is: fMRI data from OpenNeuro/HCP or R2 storage → preprocessing and representation transforms → MAE model tokenization/masking → encoder/decoder inference or training → embeddings/reconstructions → downstream evaluation in Brainmarks or inspection via visualization tools. The main boundaries are between data ingestion, model core, inference/training orchestration, and external benchmarking services.
Main entry points: models_mae.py MAE models · inference.py Inference · __init__.py Public API · main_pretrain.py Pretrain entry
Components 18
CortexMAE core
Model architecture, masking, representations, and public API for pretrained MAE use.
- Public API · User-facing entry points such as CortexMAE and file resolution helpers.
src/cortex_mae/__init__.py - MAE models · Main masked autoencoder model definitions and assembly.
src/cortex_mae/models_mae.py - Core modules · Reusable model blocks used by the MAE encoder/decoder stack.
src/cortex_mae/modules.py - Masking · Mask selection and patch dropping strategy for MAE training and inference.
src/cortex_mae/masking.py - NISC · Cortex-specific input representation and tokenization logic.
src/cortex_mae/nisc.py - Data prep · Data loading and tensor preparation for fMRI inputs.
src/cortex_mae/data.py - Transforms · Representation and preprocessing transforms for parcel, flat, and volume inputs.
src/cortex_mae/transforms.py - Utilities · Shared helpers used across preprocessing and runtime paths.
src/cortex_mae/utils.py - Flat PCA · Precomputed projection used by the flat-map input pipeline.
src/cortex_mae/resources/flat_pca_n8_224_560.npz
Training and inference
Config-driven pretraining, embedding generation, and visualization support.
- Pretrain entry · Executable entry point for config-driven masked autoencoder training.
src/cortex_mae/main_pretrain.py - Pretrain config · Default training settings and overridable hyperparameters.
src/cortex_mae/config/default_pretrain.yaml - Inference · Loads inputs and produces embeddings or related outputs from fMRI files.
src/cortex_mae/inference.py - Visualization · Inspection and visualization helpers for model outputs.
src/cortex_mae/visualization.py
Brainmarks integration
Adapter layer that plugs CortexMAE encoders into the Brainmarks probing framework.
- Brainmarks wrapper · Adapts CortexMAE encoders to the Brainmarks model interface.
src/brainmarks/models/cortex_mae_wrapper.py - Wrapper test · Checks compatibility of the CortexMAE Brainmarks integration.
src/brainmarks/models/test_cortex_mae_wrapper.py
Packaging and docs
Distribution metadata, CI release workflow, and interactive usage examples.
- README · Primary usage and architecture documentation.
README.md - Quickstart · Interactive end-to-end example for loading pretrained models and running embeddings.
notebooks/quickstart.ipynb - Publish CI · Automated packaging and publishing workflow.
.github/workflows/publish.yml
Connections 21
- Public API → MAE models: exports · Public API exposes the pretrained model entry points.
- Public API → Inference: uses · Convenience loading and embedding flow routes into inference.
- Public API → Utilities: helpers · Shared runtime helpers are surfaced through the package boundary.
- MAE models → Core modules: builds from · Model assembly depends on reusable transformer blocks.
- MAE models → Masking: applies · Masked autoencoding relies on token masking strategy.
- MAE models → NISC: encodes · Cortex-specific token representations feed the MAE model.
- MAE models → Transforms: consumes · Model inputs are prepared by preprocessing transforms.
- Data prep → Transforms: loads · Data pipeline applies representation-specific transforms.
- Transforms → Flat PCA: projects via · Flat input path uses the packaged PCA projection resource.
- Pretrain entry → Pretrain config: reads · Training is configured by the external YAML defaults.
- Pretrain entry → Data prep: builds · Training entry point constructs the fmri data pipeline.
- Pretrain entry → MAE models: instantiates · Pretraining creates the MAE architecture from configuration.
- Inference → MAE models: runs · Inference loads pretrained models to produce embeddings.
- Inference → Data prep: ingests · Runtime accepts prepared fMRI data or files through the data layer.
- Inference → Visualization: supports · Outputs can be inspected with visualization helpers.
- Brainmarks wrapper → MAE models: adapts · Brainmarks integration wraps CortexMAE encoders.
- Wrapper test → Brainmarks wrapper: verifies · Integration tests cover the adapter boundary.
- README → Public API: documents · README presents the public loading and embedding API.
- README → Pretrain entry: documents · README describes the training entry point and configuration flow.
- Quickstart → Inference: demonstrates · Quickstart notebook mirrors the embedding workflow.
- Publish CI → Public API: packages · Release workflow publishes the Python package surface.