How brain-score/vision works
Key architectural components
• Core Package: The central code is in the “brainscore_vision” directory. This package contains submodules for benchmarks, data handling, metrics, model helpers, model interface definitions, submission endpoints, and utilities.
• Benchmark Engine: The “benchmarks” subdirectory organizes different experimental benchmarks (e.g. “MajajHong2015public”, “Baker2022”, etc.). This is where benchmark definitions, testing scripts, and configuration for running experiments are defined.
• Data Layer: The “data” directory holds code for assembling, pre-processing, and packaging experimental data. It is the input source to the benchmark engine.
• Metrics: The “metrics” module contains various scoring methods (e.g., accuracy, error consistency, distribution similarity) that quantify the match between the model’s output and brain data.
• Model Helpers & Interfaces: The “model_helpers” folder (with submodules for activations, temporal processing, brain transformation, etc.) supports model evaluation. The “model_interface.py” file defines the required abstraction (BrainModel interface) that candidate models must implement.
• Submission and Integration: The “submission” folder and accompanying scripts (and files in the root such as “setup.py”, “pyproject.toml”, etc.) provide the means to package models, integrate with caching (see the “result_caching” references in README), and expose endpoints for model submission and testing.
• External Integration and CI/CD: The “.github” directory contains workflow configurations (e.g. for automerge, version bumping, integration testing) and continuous integration scripts that ensure integrity over the repository.
• Documentation & Examples: The “docs” and “examples” directories serve external documentation (and live notebooks) that outline usage, integration examples, and tutorials on scoring models.
• Testing: The “tests” directory has unit and integration tests in separate subfolders for benchmarking, metric helpers, model helpers, submission endpoints, etc.
Relationships and data flows between components
• The Data Providers module extracts and formats experimental stimuli and neural recordings (from the “data” folder) and passes these to the benchmark definitions.
• The Benchmark Engine (within “benchmarks”) orchestrates the evaluation by loading a candidate model (which adheres to “BrainModel”) via the Model Interface and then running the model on prepared input data.
• The model’s output is “post-processed” using Model Helpers (activations, temporal processing) before being compared against experimental data with Metrics modules (accuracy, CKA, etc.).
• Intermediate results, scores, and activations are cached (using the result caching mechanism configured by environment variables such as RESULTCACHING_HOME) to avoid redundant computations.
• There is a submission pipeline (in “submission”) that possibly integrates with an external website (brain-score.org) for full benchmark evaluation.
• The CI/CD pipelines (under “.github/workflows”) ensure that updates in any of these components are automatically tested and documented.
• Documentation and examples act as the user-facing component that explains how full-stack benchmarking is performed (linking conceptual elements to code examples).
Relevant technologies and design patterns
• The project is implemented in Python (requiring Python 3.11+ as specified).
• It leverages a modular design where each functional domain (data, processing, metrics) is cleanly separated.
• The caching pattern (using a dedicated result caching library) is essential to optimize repeated benchmark evaluations.
• The use of interfaces (BrainModel interface) ensures extensibility: new models need only abide by the standard interface.
• CI/CD and version management scripts ensure reproducibility and continuous integration of benchmarks and models.