How scope-ml/scope-ml works
1. What this repo is
SCoPe is an astronomy ML pipeline for classifying variable-source light curves from ZTF and Rubin/LSST-like data. The repository is split into a reusable Python package plus a large set of CLI tools, HPC batch scripts, docs, and validation data.
2. Core package boundary
The scope/ package is the library layer. It contains the classification model abstractions, taxonomy/hierarchy logic, prior handling, survey-specific data adapters, and utility code. This is the code other scripts build on; the tools/ directory is mostly orchestration around it.
3. Model layer
The main model families are represented explicitly:
- scope/nn.py, scope/nn_pytorch.py, scope/nn_applecider.py for neural-network-based classifiers
- scope/xgb.py for gradient-boosted tree models
- scope/models.py and scope/scope_class.py for shared model/classification interfaces
This suggests a common classification API with multiple backends rather than one monolithic model.
4. Taxonomy and class hierarchy
Classification is not flat. scope/hierarchy.py, scope/hierarchy_config.yaml, scope/class_priors.py, and tools/scope_taxonomy.yaml define a hierarchical label system and class priors. That hierarchy is a key architectural boundary because both training and inference must map predictions into the same taxonomy.
5. Survey/data-source adapters
The scope/surveys/ package isolates external source systems:
- kowalski.py for ZTF/Kowalski access
- fritz.py for Fritz integration
- rubin.py for Rubin DP1 access
- wise.py for WISE-related data
This is where the repo crosses from internal ML logic into survey-specific retrieval and identifier handling.
6. Feature-generation pipeline
Feature extraction is a separate subsystem under tools/featureGeneration/. It includes light-curve statistics, alert statistics, cadence/alias handling, cross-matching, and period search. The README highlights period-finding methods such as Conditional Entropy, AOV, Lomb-Scargle, and FPW, plus Fourier decomposition via periodfind and Fortran-backed AOV code in pyaov/. Feature generation is the main data-processing stage before model inference.
7. Data flow through the system
The typical pipeline is: survey/object selection → light-curve retrieval → feature generation → model training or inference → prediction outputs. Features and predictions are written in columnar/tabular formats, especially Parquet for Rubin workflows and CSV for golden/test datasets. The repo also includes mapper JSON files to translate IDs and taxonomy labels across systems.
8. Training and inference orchestration
The tools/ directory provides the operational entry points for training and inference, including job-submission wrappers and local runners. This layer handles selecting data, generating features, training algorithms, running inference, combining chunked outputs, and downloading/uploading classifications. It is the main “application” surface for end users.
9. HPC and batch-processing layer
The hpc_files/ tree contains cluster-specific SLURM templates, logs directories, and configuration for large-scale runs. There are separate workflows for:
- DNN training/inference
- XGB training/inference
- feature generation
- combining predictions
- dataset downloads and source generation
This indicates the repo is designed for distributed batch processing, not just local interactive use.
10. Rubin DP1 local-parquet path
A notable architectural branch is Rubin support with a local parquet backend. When configured with data_path, the pipeline bypasses TAP and uses local Object, ForcedSource, and Visit parquet files through a local client path. That creates a second data-access mode: remote survey API vs local bulk-file processing.
11. Validation and reference data
The data/golden/ files and tests/ directory provide regression coverage for class-specific outputs and critical functionality such as neural-network behavior, period search, Rubin support, and WISE support. These are important architectural anchors because they constrain model and feature changes.
12. Docs and operational knowledge
The documentation is substantial and mirrors the code structure: getting started, feature generation, training, inference, Rubin-specific workflows, HPC scripts, and field guides for classes. For an architecture diagram, the docs are effectively part of the system design because they define the supported workflows and how the subsystems are intended to be composed.
Main entry points: tests Tests · featureGeneration Feature gen · generate_features.py Generate features · regenerate_training_set.py Training
Components 22
Library
Reusable Python package with shared model, taxonomy, and survey-adapter logic.
- Models · Shared classification interfaces and model abstractions.
scope/models.py - Scope class · Main classification object and workflow-facing model wrapper.
scope/scope_class.py - Hierarchy · Hierarchical label system used across training and inference.
scope/hierarchy.py - Priors · Class prior handling and prior-driven adjustments.
scope/class_priors.py - Neural net · Neural-network classifier entrypoint.
scope/nn.py - PyTorch NN · PyTorch implementation of the neural classifier.
scope/nn_pytorch.py - Applecider NN · Alternative neural-network backend.
scope/nn_applecider.py - XGBoost · Gradient-boosted tree classifier backend.
scope/xgb.py - Survey adapters · Survey-specific retrieval and identifier integration.
scope/surveys/__init__.py - Rubin · Rubin DP1 and local-parquet access path.
scope/surveys/rubin.py - Kowalski · ZTF/Kowalski access and object lookup.
scope/surveys/kowalski.py - Fritz · Fritz integration for source and annotation workflows.
scope/surveys/fritz.py - WISE · WISE-related data access and support logic.
scope/surveys/wise.py
Pipeline
Feature generation and end-to-end training/inference orchestration.
- Feature gen · Feature-extraction subsystem for light curves and alerts.
tools/featureGeneration - Period search · Period-finding algorithms and Fourier/AOV-based analysis.
tools/featureGeneration/periodsearch.py - Generate features · Main feature-generation entrypoint and workflow driver.
tools/generate_features.py - Inference · Classification and prediction generation entrypoint.
tools/inference.py - Training · Model-training entrypoint and dataset preparation driver.
tools/regenerate_training_set.py
Ops
HPC batch templates, scripts, and workflow helpers.
- HPC scripts · Cluster templates and job scripts for large-scale runs.
hpc_files
Docs & Validation
Reference docs, golden data, and tests that anchor supported behavior.
- Docs · User guides and reference material mirroring the code structure.
docs - Golden data · Reference CSV outputs used to validate key classifications.
data/golden - Tests · Regression tests for model backends, period search, Rubin, and WISE.
tests
Connections 32
- Models → Scope class: implements · Shared interfaces are wrapped by the main classification object.
- Models → Neural net: backend · Model abstractions are realized by the neural-network family.
- Models → XGBoost: backend · Model abstractions are also realized by boosted-tree classifiers.
- Neural net → PyTorch NN: variant · Primary neural backend is implemented in PyTorch.
- Neural net → Applecider NN: variant · Alternative neural backend for specialized runs.
- Scope class → Hierarchy: uses · Classification objects resolve predictions through the label hierarchy.
- Hierarchy → Priors: shapes · Hierarchy and priors jointly define prediction weighting.
- Survey adapters → Kowalski: includes · Survey package exposes ZTF/Kowalski integration.
- Survey adapters → Fritz: includes · Survey package exposes Fritz integration.
- Survey adapters → Rubin: includes · Survey package exposes Rubin access paths.
- Survey adapters → WISE: includes · Survey package exposes WISE support.
- Rubin → Feature gen: feeds · Rubin objects and forced sources are converted into features.
- Kowalski → Feature gen: feeds · Survey retrieval provides light curves for feature extraction.
- Fritz → Generate features: orchestrates · Fritz-based downloads can drive feature generation workflows.
- Feature gen → Period search: contains · Period-search is a major internal stage of feature extraction.
- Feature gen → Generate features: drives · Feature extraction is executed through the generation entrypoint.
- Generate features → Training: prepares · Generated features are used to build training datasets.
- Generate features → Inference: prepares · Generated features are also consumed for prediction runs.
- Training → Neural net: trains · Training workflows fit neural classifier backends.
- Training → XGBoost: trains · Training workflows fit boosted-tree backends.
- Inference → Scope class: runs · Inference executes the shared classification API.
- HPC scripts → Training: batch · SLURM templates and scripts launch training jobs.
- HPC scripts → Inference: batch · SLURM templates and scripts launch inference jobs.
- HPC scripts → Generate features: batch · Cluster jobs also run feature generation at scale.
- Docs → Feature gen: documents · User guides describe feature-generation workflows.
- Docs → Training: documents · Docs explain model training and dataset regeneration.
- Docs → Inference: documents · Docs describe prediction and upload/download workflows.
- Golden data → Tests: anchors · Golden CSVs provide reference outputs for regression checks.
- Tests → PyTorch NN: verifies · Tests cover neural backend behavior.
- Tests → Rubin: verifies · Tests cover Rubin/local-parquet behavior.
- Tests → WISE: verifies · Tests cover WISE support.
- Tests → Period search: verifies · Tests cover period-search routines.