# medarc-ai/cortexmae architecture

> Architecture diagram of the GitHub repository medarc-ai/cortexmae, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/medarc-ai/cortexmae
- Repository: https://github.com/medarc-ai/cortexmae

## Overview

1. **Repository purpose and top-level boundary**
   CortexMAE is a Python package that provides pretrained masked autoencoder models for fMRI, plus the training and inference code needed to reproduce them. The repo has two main package roots: `cortex_mae` for the model system itself, and `brainmarks.models` for integration into the external Brainmarks probing framework.

2. **Core model subsystem**
   The heart of the repo lives in `src/cortex_mae/models_mae.py`, `modules.py`, `masking.py`, and `nisc.py`. This layer implements the MAE-style architecture: patch/token construction, masking strategy, transformer encoder/decoder components, and the Cortex-specific input representations. These files form the model boundary that everything else feeds into.

3. **Input representation and preprocessing**
   The model supports three input spaces called out in the README: parcel, flat, and volume. The preprocessing path is handled by `data.py`, `transforms.py`, and `utils.py`, which prepare fMRI time series into model-ready tensors. The flat-map representation also depends on the packaged resource `resources/flat_pca_n8_224_560.npz`, indicating a learned or fixed projection used in the flat input pipeline.

4. **Training pipeline**
   `main_pretrain.py` is the executable entry point for pretraining. It reads settings from `config/default_pretrain.yaml`, builds the data pipeline, instantiates the model, and runs masked autoencoder training. The architecture here is “config-driven training”: defaults are externalized in YAML, and command-line overrides can switch input space, learning rate, or model variant without changing code.

5. **Inference and embedding generation**
   `inference.py` exposes the runtime path for turning an fMRI file into embeddings, which is what the README’s `CortexMAE.from_pretrained(...).run_embedding(...)` flow uses. This is the primary inference boundary: raw or preprocessed fMRI enters, and patch-level embeddings or related outputs leave. `visualization.py` likely supports interpreting those embeddings or reconstructions.

6. **Pretrained model distribution**
   The repo is designed to consume pretrained weights rather than only train from scratch. The README references HuggingFace-hosted checkpoints, and the package exposes model loading helpers via the public API in `src/cortex_mae/__init__.py`. This means the runtime architecture includes an external model registry/service boundary, not just local checkpoints.

7. **Data sources and storage boundaries**
   Two distinct data infrastructures appear in the architecture: HCP-YA pretraining data stored as webdataset shards on R2, and benchmark datasets stored as HuggingFace Arrow format on the MedARC R2 bucket via Brainmarks. The repo therefore straddles both streamed training data and separately managed downstream evaluation datasets, with AWS/R2 credentials required for access.

8. **Brainmarks integration for downstream evaluation**
   `src/brainmarks/models/cortex_mae_wrapper.py` adapts CortexMAE encoders to the Brainmarks API. This is a deliberate boundary layer: CortexMAE owns model construction and embeddings, while Brainmarks owns probe evaluation, task selection, and benchmarking orchestration. The README’s `brainmarks.main_probe` invocation reflects this split.

9. **Public package surface**
   `src/cortex_mae/__init__.py` defines the user-facing API, centered on `CortexMAE` and helpers like `resolve_file`. This is the main stable entry point for consumers who want to load pretrained checkpoints, resolve remote files, and run embeddings without touching the internal training stack.

10. **Tests as subsystem coverage**
   The test suite is organized around the main architectural units: `test_models.py`, `test_modules.py`, `test_transforms.py`, and `test_inference.py`. There is also a wrapper test in `brainmarks/models/test_cortex_mae_wrapper.py`, which helps confirm the integration boundary remains compatible with Brainmarks.

11. **Packaging and release workflow**
   `pyproject.toml`, `MANIFEST.in`, and the GitHub Actions workflow in `.github/workflows/publish.yml` show that this is a distributable Python package with automated publishing. The repo also includes `uv.lock` and README installation instructions for `uv`, so reproducible environment management is part of the delivery path.

12. **Documentation and usage examples**
   `notebooks/quickstart.ipynb` is the primary interactive demo and mirrors the README’s inference flow. It functions as a reference integration layer rather than core logic, showing how the package is intended to be used end-to-end with pretrained weights and remote fMRI files.

13. **Overall data flow**
   End-to-end, the architecture is: fMRI data from OpenNeuro/HCP or R2 storage → preprocessing and representation transforms → MAE model tokenization/masking → encoder/decoder inference or training → embeddings/reconstructions → downstream evaluation in Brainmarks or inspection via visualization tools. The main boundaries are between data ingestion, model core, inference/training orchestration, and external benchmarking services.

Main entry points: [`src/cortex_mae/models_mae.py`](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/models_mae.py) (MAE models), [`src/cortex_mae/inference.py`](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/inference.py) (Inference), [`src/cortex_mae/__init__.py`](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/__init__.py) (Public API), [`src/cortex_mae/main_pretrain.py`](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/main_pretrain.py) (Pretrain entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["CortexMAE core"]
  node_api["Public API<br/>package init<br/>[__init__.py]"]
  node_models["MAE models<br/>architecture<br/>[models_mae.py]"]
  node_modules["Core modules<br/>layers<br/>[modules.py]"]
  node_masking["Masking<br/>[masking.py]"]
  node_nisc["NISC<br/>representation<br/>[nisc.py]"]
  node_data["Data prep<br/>fmri pipeline<br/>[data.py]"]
  node_transforms["Transforms<br/>preprocess<br/>[transforms.py]"]
  node_utils["Utilities<br/>helpers<br/>[utils.py]"]
  node_flat_resource[("Flat PCA<br/>packed resource")]
end

subgraph group_pipeline["Training and inference"]
  node_pretrain["Pretrain entry<br/>training cli<br/>[main_pretrain.py]"]
  node_pretrain_cfg["Pretrain config<br/>yaml config"]
  node_inference["Inference<br/>embedding runtime<br/>[inference.py]"]
  node_viz["Visualization<br/>analysis<br/>[visualization.py]"]
end

subgraph group_integration["Brainmarks integration"]
  node_wrapper["Brainmarks wrapper<br/>adapter"]
  node_brainmarks_test["Wrapper test"]
end

subgraph group_packaging["Packaging and docs"]
  node_readme["README<br/>docs<br/>[README.md]"]
  node_notebook["Quickstart<br/>example notebook<br/>[quickstart.ipynb]"]
  node_publish["Publish CI<br/>release workflow<br/>[publish.yml]"]
end

node_api -->|"exports"| node_models
node_api -->|"uses"| node_inference
node_api -->|"helpers"| node_utils
node_models -->|"builds from"| node_modules
node_models -->|"applies"| node_masking
node_models -->|"encodes"| node_nisc
node_models -->|"consumes"| node_transforms
node_data -->|"loads"| node_transforms
node_transforms -->|"projects via"| node_flat_resource
node_pretrain -->|"reads"| node_pretrain_cfg
node_pretrain -->|"builds"| node_data
node_pretrain -->|"instantiates"| node_models
node_inference -->|"runs"| node_models
node_inference -->|"ingests"| node_data
node_inference -->|"supports"| node_viz
node_wrapper -->|"adapts"| node_models
node_brainmarks_test -->|"verifies"| node_wrapper
node_readme -->|"documents"| node_api
node_readme -->|"documents"| node_pretrain
node_notebook -->|"demonstrates"| node_inference
node_publish -->|"packages"| node_api

click node_api "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/__init__.py"
click node_models "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/models_mae.py"
click node_modules "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/modules.py"
click node_masking "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/masking.py"
click node_nisc "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/nisc.py"
click node_data "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/data.py"
click node_transforms "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/transforms.py"
click node_utils "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/utils.py"
click node_flat_resource "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/resources/flat_pca_n8_224_560.npz"
click node_pretrain "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/main_pretrain.py"
click node_pretrain_cfg "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/config/default_pretrain.yaml"
click node_inference "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/inference.py"
click node_viz "https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/visualization.py"
click node_wrapper "https://github.com/medarc-ai/cortexmae/blob/main/src/brainmarks/models/cortex_mae_wrapper.py"
click node_brainmarks_test "https://github.com/medarc-ai/cortexmae/blob/main/src/brainmarks/models/test_cortex_mae_wrapper.py"
click node_readme "https://github.com/medarc-ai/cortexmae/blob/main/README.md"
click node_notebook "https://github.com/medarc-ai/cortexmae/blob/main/notebooks/quickstart.ipynb"
click node_publish "https://github.com/medarc-ai/cortexmae/blob/main/.github/workflows/publish.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_models,node_modules,node_masking,node_nisc,node_data,node_transforms,node_utils,node_flat_resource toneBlue
class node_pretrain,node_pretrain_cfg,node_inference,node_viz toneAmber
class node_wrapper,node_brainmarks_test toneMint
class node_readme,node_notebook,node_publish toneRose
```

## Components

### CortexMAE core

Model architecture, masking, representations, and public API for pretrained MAE use.

- [Public API](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/__init__.py): package init · User-facing entry points such as CortexMAE and file resolution helpers. · `src/cortex_mae/__init__.py`
- [MAE models](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/models_mae.py): architecture · Main masked autoencoder model definitions and assembly. · `src/cortex_mae/models_mae.py`
- [Core modules](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/modules.py): layers · Reusable model blocks used by the MAE encoder/decoder stack. · `src/cortex_mae/modules.py`
- [Masking](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/masking.py): token masking · Mask selection and patch dropping strategy for MAE training and inference. · `src/cortex_mae/masking.py`
- [NISC](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/nisc.py): representation · Cortex-specific input representation and tokenization logic. · `src/cortex_mae/nisc.py`
- [Data prep](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/data.py): fmri pipeline · Data loading and tensor preparation for fMRI inputs. · `src/cortex_mae/data.py`
- [Transforms](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/transforms.py): preprocess · Representation and preprocessing transforms for parcel, flat, and volume inputs. · `src/cortex_mae/transforms.py`
- [Utilities](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/utils.py): helpers · Shared helpers used across preprocessing and runtime paths. · `src/cortex_mae/utils.py`
- [Flat PCA](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/resources/flat_pca_n8_224_560.npz): packed resource · Precomputed projection used by the flat-map input pipeline. · `src/cortex_mae/resources/flat_pca_n8_224_560.npz`

### Training and inference

Config-driven pretraining, embedding generation, and visualization support.

- [Pretrain entry](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/main_pretrain.py): training cli · Executable entry point for config-driven masked autoencoder training. · `src/cortex_mae/main_pretrain.py`
- [Pretrain config](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/config/default_pretrain.yaml): yaml config · Default training settings and overridable hyperparameters. · `src/cortex_mae/config/default_pretrain.yaml`
- [Inference](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/inference.py): embedding runtime · Loads inputs and produces embeddings or related outputs from fMRI files. · `src/cortex_mae/inference.py`
- [Visualization](https://github.com/medarc-ai/cortexmae/blob/main/src/cortex_mae/visualization.py): analysis · Inspection and visualization helpers for model outputs. · `src/cortex_mae/visualization.py`

### Brainmarks integration

Adapter layer that plugs CortexMAE encoders into the Brainmarks probing framework.

- [Brainmarks wrapper](https://github.com/medarc-ai/cortexmae/blob/main/src/brainmarks/models/cortex_mae_wrapper.py): adapter · Adapts CortexMAE encoders to the Brainmarks model interface. · `src/brainmarks/models/cortex_mae_wrapper.py`
- [Wrapper test](https://github.com/medarc-ai/cortexmae/blob/main/src/brainmarks/models/test_cortex_mae_wrapper.py): test · Checks compatibility of the CortexMAE Brainmarks integration. · `src/brainmarks/models/test_cortex_mae_wrapper.py`

### Packaging and docs

Distribution metadata, CI release workflow, and interactive usage examples.

- [README](https://github.com/medarc-ai/cortexmae/blob/main/README.md): docs · Primary usage and architecture documentation. · `README.md`
- [Quickstart](https://github.com/medarc-ai/cortexmae/blob/main/notebooks/quickstart.ipynb): example notebook · Interactive end-to-end example for loading pretrained models and running embeddings. · `notebooks/quickstart.ipynb`
- [Publish CI](https://github.com/medarc-ai/cortexmae/blob/main/.github/workflows/publish.yml): release workflow · Automated packaging and publishing workflow. · `.github/workflows/publish.yml`

## Connections

- Public API → MAE models: exports · Public API exposes the pretrained model entry points.
- Public API → Inference: uses · Convenience loading and embedding flow routes into inference.
- Public API → Utilities: helpers · Shared runtime helpers are surfaced through the package boundary.
- MAE models → Core modules: builds from · Model assembly depends on reusable transformer blocks.
- MAE models → Masking: applies · Masked autoencoding relies on token masking strategy.
- MAE models → NISC: encodes · Cortex-specific token representations feed the MAE model.
- MAE models → Transforms: consumes · Model inputs are prepared by preprocessing transforms.
- Data prep → Transforms: loads · Data pipeline applies representation-specific transforms.
- Transforms → Flat PCA: projects via · Flat input path uses the packaged PCA projection resource.
- Pretrain entry → Pretrain config: reads · Training is configured by the external YAML defaults.
- Pretrain entry → Data prep: builds · Training entry point constructs the fmri data pipeline.
- Pretrain entry → MAE models: instantiates · Pretraining creates the MAE architecture from configuration.
- Inference → MAE models: runs · Inference loads pretrained models to produce embeddings.
- Inference → Data prep: ingests · Runtime accepts prepared fMRI data or files through the data layer.
- Inference → Visualization: supports · Outputs can be inspected with visualization helpers.
- Brainmarks wrapper → MAE models: adapts · Brainmarks integration wraps CortexMAE encoders.
- Wrapper test → Brainmarks wrapper: verifies · Integration tests cover the adapter boundary.
- README → Public API: documents · README presents the public loading and embedding API.
- README → Pretrain entry: documents · README describes the training entry point and configuration flow.
- Quickstart → Inference: demonstrates · Quickstart notebook mirrors the embedding workflow.
- Publish CI → Public API: packages · Release workflow publishes the Python package surface.
