How epics-lab/brainfmbench works
BrainFMBench is a structural-MRI foundation-model benchmark: submissions provide metadata and feature extractors (or precomputed features), extraction runs on a compute cluster, and downstream probes score features against shared labels for the leaderboard. The graph preserves the documented preprocessing, extraction, scoring, and publishing stages, plus cluster orchestration and representative model implementations. README-documented workflow edges are cited there; source excerpts confirm the runner-to-SSH dependency and the scoring script’s use of the scorer. Other model implementations are grouped where their source was not sampled.
Main entry points: score_all.py Scoring orchestration · models Feature vectors · preprocess.py Brain2Vec alignment · scorer.py Downstream probes
Components 19
Submissions
- Model submission
models - Example submission
example-submission/extract.py
Feature computation
- MRI preprocessing
- Brain2Vec alignment
models/brain2vec/preprocess.py - Brain2Vec extractor
models/brain2vec/extract.py - Dinov3D extractor
models/dinov3d-neuro/extract.py - un-CNN extractor
models/uncnn/extract.py - Other model extractors
models
Evaluation
- Feature vectors
models - Task labels
labels - Scoring orchestration
scripts/score_all.py - Downstream probes
eval/scorer.py - Model results
models
Benchmark operations
- Cluster runner
cluster/cluster_runner.py - Cluster SSH
cluster/ssh_manager.py - Leaderboard
scripts/make_leaderboard.py
Other components
- Contributor
- Compute cluster
- GitHub Actions
Connections 20
- Contributor → Model submission: submits model
README.md - Model submission → Cluster runner: queues extraction
README.md - Cluster runner → Cluster SSH: uses SSH
cluster/cluster_runner.py - Cluster SSH → Compute cluster: stages and commands
cluster/ssh_manager.py - MRI preprocessing → Compute cluster: prepares MRI data
README.md - Compute cluster → Brain2Vec alignment: runs model preprocessing
- Brain2Vec alignment → Brain2Vec extractor: aligns extractor input
models/brain2vec/preprocess.py - Compute cluster → Brain2Vec extractor: runs extraction
README.md - Compute cluster → Dinov3D extractor: runs extraction
README.md - Compute cluster → un-CNN extractor: runs extraction
README.md - Compute cluster → Other model extractors: runs extraction
README.md - Example submission → Feature vectors: writes embeddings
example-submission/extract.py - Compute cluster → Feature vectors: returns extracted vectors
README.md - Feature vectors → Scoring orchestration: supplies features
scripts/score_all.py - Task labels → Scoring orchestration: supplies labels
scripts/score_all.py - Scoring orchestration → Downstream probes: calls scorer
scripts/score_all.py - Downstream probes → Model results: produces scores
eval/scorer.py - Model results → Leaderboard: feeds leaderboard
README.md - GitHub Actions → Scoring orchestration: scores features
README.md - GitHub Actions → Leaderboard: updates ranking
README.md