# scope-ml/scope-ml architecture

> Architecture diagram of the GitHub repository scope-ml/scope-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scope-ml/scope-ml
- Repository: https://github.com/scope-ml/scope-ml

## Overview

1. **What this repo is**
   SCoPe is an astronomy ML pipeline for classifying variable-source light curves from ZTF and Rubin/LSST-like data. The repository is split into a reusable Python package plus a large set of CLI tools, HPC batch scripts, docs, and validation data.

2. **Core package boundary**
   The `scope/` package is the library layer. It contains the classification model abstractions, taxonomy/hierarchy logic, prior handling, survey-specific data adapters, and utility code. This is the code other scripts build on; the `tools/` directory is mostly orchestration around it.

3. **Model layer**
   The main model families are represented explicitly:
   - `scope/nn.py`, `scope/nn_pytorch.py`, `scope/nn_applecider.py` for neural-network-based classifiers
   - `scope/xgb.py` for gradient-boosted tree models
   - `scope/models.py` and `scope/scope_class.py` for shared model/classification interfaces  
   This suggests a common classification API with multiple backends rather than one monolithic model.

4. **Taxonomy and class hierarchy**
   Classification is not flat. `scope/hierarchy.py`, `scope/hierarchy_config.yaml`, `scope/class_priors.py`, and `tools/scope_taxonomy.yaml` define a hierarchical label system and class priors. That hierarchy is a key architectural boundary because both training and inference must map predictions into the same taxonomy.

5. **Survey/data-source adapters**
   The `scope/surveys/` package isolates external source systems:
   - `kowalski.py` for ZTF/Kowalski access
   - `fritz.py` for Fritz integration
   - `rubin.py` for Rubin DP1 access
   - `wise.py` for WISE-related data  
   This is where the repo crosses from internal ML logic into survey-specific retrieval and identifier handling.

6. **Feature-generation pipeline**
   Feature extraction is a separate subsystem under `tools/featureGeneration/`. It includes light-curve statistics, alert statistics, cadence/alias handling, cross-matching, and period search. The README highlights period-finding methods such as Conditional Entropy, AOV, Lomb-Scargle, and FPW, plus Fourier decomposition via `periodfind` and Fortran-backed AOV code in `pyaov/`. Feature generation is the main data-processing stage before model inference.

7. **Data flow through the system**
   The typical pipeline is: survey/object selection → light-curve retrieval → feature generation → model training or inference → prediction outputs. Features and predictions are written in columnar/tabular formats, especially Parquet for Rubin workflows and CSV for golden/test datasets. The repo also includes mapper JSON files to translate IDs and taxonomy labels across systems.

8. **Training and inference orchestration**
   The `tools/` directory provides the operational entry points for training and inference, including job-submission wrappers and local runners. This layer handles selecting data, generating features, training algorithms, running inference, combining chunked outputs, and downloading/uploading classifications. It is the main “application” surface for end users.

9. **HPC and batch-processing layer**
   The `hpc_files/` tree contains cluster-specific SLURM templates, logs directories, and configuration for large-scale runs. There are separate workflows for:
   - DNN training/inference
   - XGB training/inference
   - feature generation
   - combining predictions
   - dataset downloads and source generation  
   This indicates the repo is designed for distributed batch processing, not just local interactive use.

10. **Rubin DP1 local-parquet path**
   A notable architectural branch is Rubin support with a local parquet backend. When configured with `data_path`, the pipeline bypasses TAP and uses local `Object`, `ForcedSource`, and `Visit` parquet files through a local client path. That creates a second data-access mode: remote survey API vs local bulk-file processing.

11. **Validation and reference data**
   The `data/golden/` files and `tests/` directory provide regression coverage for class-specific outputs and critical functionality such as neural-network behavior, period search, Rubin support, and WISE support. These are important architectural anchors because they constrain model and feature changes.

12. **Docs and operational knowledge**
   The documentation is substantial and mirrors the code structure: getting started, feature generation, training, inference, Rubin-specific workflows, HPC scripts, and field guides for classes. For an architecture diagram, the docs are effectively part of the system design because they define the supported workflows and how the subsystems are intended to be composed.

Main entry points: [`tests`](https://github.com/scope-ml/scope-ml/tree/main/tests) (Tests), [`tools/featureGeneration`](https://github.com/scope-ml/scope-ml/tree/main/tools/featureGeneration) (Feature gen), [`tools/generate_features.py`](https://github.com/scope-ml/scope-ml/blob/main/tools/generate_features.py) (Generate features), [`tools/regenerate_training_set.py`](https://github.com/scope-ml/scope-ml/blob/main/tools/regenerate_training_set.py) (Training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_library["Library"]
  node_scope_models["Models<br/>api layer<br/>[models.py]"]
  node_scope_scope_class["Scope class<br/>core API<br/>[scope_class.py]"]
  node_scope_hierarchy["Hierarchy<br/>taxonomy<br/>[hierarchy.py]"]
  node_scope_class_priors["Priors<br/>calibration<br/>[class_priors.py]"]
  node_scope_nn["Neural net<br/>model backend<br/>[nn.py]"]
  node_scope_nn_pytorch["PyTorch NN<br/>model backend<br/>[nn_pytorch.py]"]
  node_scope_nn_applecider["Applecider NN<br/>model backend<br/>[nn_applecider.py]"]
  node_scope_xgb["XGBoost<br/>model backend<br/>[xgb.py]"]
  node_scope_surveys["Survey adapters<br/>data access<br/>[__init__.py]"]
  node_surveys_rubin["Rubin<br/>survey adapter<br/>[rubin.py]"]
  node_surveys_kowalski["Kowalski<br/>survey adapter<br/>[kowalski.py]"]
  node_surveys_fritz["Fritz<br/>survey adapter<br/>[fritz.py]"]
  node_surveys_wise["WISE<br/>survey adapter<br/>[wise.py]"]
end

subgraph group_group_pipeline["Pipeline"]
  node_feature_generation["Feature gen<br/>pipeline stage"]
  node_periodsearch["Period search<br/>feature stage<br/>[periodsearch.py]"]
  node_generate_features["Generate features<br/>orchestration"]
  node_inference["Inference<br/>orchestration<br/>[inference.py]"]
  node_training["Training<br/>orchestration"]
end

subgraph group_group_ops["Ops"]
  node_hpc_files["HPC scripts<br/>batch ops"]
end

subgraph group_group_docs["Docs & Validation"]
  node_docs["Docs<br/>reference"]
  node_golden_data[("Golden data<br/>validation data")]
  node_tests["Tests<br/>validation"]
end

node_scope_models -->|"implements"| node_scope_scope_class
node_scope_models -->|"backend"| node_scope_nn
node_scope_models -->|"backend"| node_scope_xgb
node_scope_nn -->|"variant"| node_scope_nn_pytorch
node_scope_nn -->|"variant"| node_scope_nn_applecider
node_scope_scope_class -->|"uses"| node_scope_hierarchy
node_scope_hierarchy -->|"shapes"| node_scope_class_priors
node_scope_surveys -->|"includes"| node_surveys_kowalski
node_scope_surveys -->|"includes"| node_surveys_fritz
node_scope_surveys -->|"includes"| node_surveys_rubin
node_scope_surveys -->|"includes"| node_surveys_wise
node_surveys_rubin -->|"feeds"| node_feature_generation
node_surveys_kowalski -->|"feeds"| node_feature_generation
node_surveys_fritz -->|"orchestrates"| node_generate_features
node_feature_generation -->|"contains"| node_periodsearch
node_feature_generation -->|"drives"| node_generate_features
node_generate_features -->|"prepares"| node_training
node_generate_features -->|"prepares"| node_inference
node_training -->|"trains"| node_scope_nn
node_training -->|"trains"| node_scope_xgb
node_inference -->|"runs"| node_scope_scope_class
node_hpc_files -->|"batch"| node_training
node_hpc_files -->|"batch"| node_inference
node_hpc_files -->|"batch"| node_generate_features
node_docs -->|"documents"| node_feature_generation
node_docs -->|"documents"| node_training
node_docs -->|"documents"| node_inference
node_golden_data -->|"anchors"| node_tests
node_tests -->|"verifies"| node_scope_nn_pytorch
node_tests -->|"verifies"| node_surveys_rubin
node_tests -->|"verifies"| node_surveys_wise
node_tests -->|"verifies"| node_periodsearch

click node_scope_models "https://github.com/scope-ml/scope-ml/blob/main/scope/models.py"
click node_scope_scope_class "https://github.com/scope-ml/scope-ml/blob/main/scope/scope_class.py"
click node_scope_hierarchy "https://github.com/scope-ml/scope-ml/blob/main/scope/hierarchy.py"
click node_scope_class_priors "https://github.com/scope-ml/scope-ml/blob/main/scope/class_priors.py"
click node_scope_nn "https://github.com/scope-ml/scope-ml/blob/main/scope/nn.py"
click node_scope_nn_pytorch "https://github.com/scope-ml/scope-ml/blob/main/scope/nn_pytorch.py"
click node_scope_nn_applecider "https://github.com/scope-ml/scope-ml/blob/main/scope/nn_applecider.py"
click node_scope_xgb "https://github.com/scope-ml/scope-ml/blob/main/scope/xgb.py"
click node_scope_surveys "https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/__init__.py"
click node_surveys_rubin "https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/rubin.py"
click node_surveys_kowalski "https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/kowalski.py"
click node_surveys_fritz "https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/fritz.py"
click node_surveys_wise "https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/wise.py"
click node_feature_generation "https://github.com/scope-ml/scope-ml/tree/main/tools/featureGeneration"
click node_periodsearch "https://github.com/scope-ml/scope-ml/blob/main/tools/featureGeneration/periodsearch.py"
click node_generate_features "https://github.com/scope-ml/scope-ml/blob/main/tools/generate_features.py"
click node_inference "https://github.com/scope-ml/scope-ml/blob/main/tools/inference.py"
click node_training "https://github.com/scope-ml/scope-ml/blob/main/tools/regenerate_training_set.py"
click node_hpc_files "https://github.com/scope-ml/scope-ml/tree/main/hpc_files"
click node_docs "https://github.com/scope-ml/scope-ml/tree/main/docs"
click node_golden_data "https://github.com/scope-ml/scope-ml/tree/main/data/golden"
click node_tests "https://github.com/scope-ml/scope-ml/tree/main/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scope_models,node_scope_scope_class,node_scope_hierarchy,node_scope_class_priors,node_scope_nn,node_scope_nn_pytorch,node_scope_nn_applecider,node_scope_xgb,node_scope_surveys,node_surveys_rubin,node_surveys_kowalski,node_surveys_fritz,node_surveys_wise toneBlue
class node_feature_generation,node_periodsearch,node_generate_features,node_inference,node_training toneAmber
class node_hpc_files toneMint
class node_docs,node_golden_data,node_tests toneRose
```

## Components

### Library

Reusable Python package with shared model, taxonomy, and survey-adapter logic.

- [Models](https://github.com/scope-ml/scope-ml/blob/main/scope/models.py): api layer · Shared classification interfaces and model abstractions. · `scope/models.py`
- [Scope class](https://github.com/scope-ml/scope-ml/blob/main/scope/scope_class.py): core API · Main classification object and workflow-facing model wrapper. · `scope/scope_class.py`
- [Hierarchy](https://github.com/scope-ml/scope-ml/blob/main/scope/hierarchy.py): taxonomy · Hierarchical label system used across training and inference. · `scope/hierarchy.py`
- [Priors](https://github.com/scope-ml/scope-ml/blob/main/scope/class_priors.py): calibration · Class prior handling and prior-driven adjustments. · `scope/class_priors.py`
- [Neural net](https://github.com/scope-ml/scope-ml/blob/main/scope/nn.py): model backend · Neural-network classifier entrypoint. · `scope/nn.py`
- [PyTorch NN](https://github.com/scope-ml/scope-ml/blob/main/scope/nn_pytorch.py): model backend · PyTorch implementation of the neural classifier. · `scope/nn_pytorch.py`
- [Applecider NN](https://github.com/scope-ml/scope-ml/blob/main/scope/nn_applecider.py): model backend · Alternative neural-network backend. · `scope/nn_applecider.py`
- [XGBoost](https://github.com/scope-ml/scope-ml/blob/main/scope/xgb.py): model backend · Gradient-boosted tree classifier backend. · `scope/xgb.py`
- [Survey adapters](https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/__init__.py): data access · Survey-specific retrieval and identifier integration. · `scope/surveys/__init__.py`
- [Rubin](https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/rubin.py): survey adapter · Rubin DP1 and local-parquet access path. · `scope/surveys/rubin.py`
- [Kowalski](https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/kowalski.py): survey adapter · ZTF/Kowalski access and object lookup. · `scope/surveys/kowalski.py`
- [Fritz](https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/fritz.py): survey adapter · Fritz integration for source and annotation workflows. · `scope/surveys/fritz.py`
- [WISE](https://github.com/scope-ml/scope-ml/blob/main/scope/surveys/wise.py): survey adapter · WISE-related data access and support logic. · `scope/surveys/wise.py`

### Pipeline

Feature generation and end-to-end training/inference orchestration.

- [Feature gen](https://github.com/scope-ml/scope-ml/tree/main/tools/featureGeneration): pipeline stage · Feature-extraction subsystem for light curves and alerts. · `tools/featureGeneration`
- [Period search](https://github.com/scope-ml/scope-ml/blob/main/tools/featureGeneration/periodsearch.py): feature stage · Period-finding algorithms and Fourier/AOV-based analysis. · `tools/featureGeneration/periodsearch.py`
- [Generate features](https://github.com/scope-ml/scope-ml/blob/main/tools/generate_features.py): orchestration · Main feature-generation entrypoint and workflow driver. · `tools/generate_features.py`
- [Inference](https://github.com/scope-ml/scope-ml/blob/main/tools/inference.py): orchestration · Classification and prediction generation entrypoint. · `tools/inference.py`
- [Training](https://github.com/scope-ml/scope-ml/blob/main/tools/regenerate_training_set.py): orchestration · Model-training entrypoint and dataset preparation driver. · `tools/regenerate_training_set.py`

### Ops

HPC batch templates, scripts, and workflow helpers.

- [HPC scripts](https://github.com/scope-ml/scope-ml/tree/main/hpc_files): batch ops · Cluster templates and job scripts for large-scale runs. · `hpc_files`

### Docs & Validation

Reference docs, golden data, and tests that anchor supported behavior.

- [Docs](https://github.com/scope-ml/scope-ml/tree/main/docs): reference · User guides and reference material mirroring the code structure. · `docs`
- [Golden data](https://github.com/scope-ml/scope-ml/tree/main/data/golden): validation data · Reference CSV outputs used to validate key classifications. · `data/golden`
- [Tests](https://github.com/scope-ml/scope-ml/tree/main/tests): validation · Regression tests for model backends, period search, Rubin, and WISE. · `tests`

## Connections

- Models → Scope class: implements · Shared interfaces are wrapped by the main classification object.
- Models → Neural net: backend · Model abstractions are realized by the neural-network family.
- Models → XGBoost: backend · Model abstractions are also realized by boosted-tree classifiers.
- Neural net → PyTorch NN: variant · Primary neural backend is implemented in PyTorch.
- Neural net → Applecider NN: variant · Alternative neural backend for specialized runs.
- Scope class → Hierarchy: uses · Classification objects resolve predictions through the label hierarchy.
- Hierarchy → Priors: shapes · Hierarchy and priors jointly define prediction weighting.
- Survey adapters → Kowalski: includes · Survey package exposes ZTF/Kowalski integration.
- Survey adapters → Fritz: includes · Survey package exposes Fritz integration.
- Survey adapters → Rubin: includes · Survey package exposes Rubin access paths.
- Survey adapters → WISE: includes · Survey package exposes WISE support.
- Rubin → Feature gen: feeds · Rubin objects and forced sources are converted into features.
- Kowalski → Feature gen: feeds · Survey retrieval provides light curves for feature extraction.
- Fritz → Generate features: orchestrates · Fritz-based downloads can drive feature generation workflows.
- Feature gen → Period search: contains · Period-search is a major internal stage of feature extraction.
- Feature gen → Generate features: drives · Feature extraction is executed through the generation entrypoint.
- Generate features → Training: prepares · Generated features are used to build training datasets.
- Generate features → Inference: prepares · Generated features are also consumed for prediction runs.
- Training → Neural net: trains · Training workflows fit neural classifier backends.
- Training → XGBoost: trains · Training workflows fit boosted-tree backends.
- Inference → Scope class: runs · Inference executes the shared classification API.
- HPC scripts → Training: batch · SLURM templates and scripts launch training jobs.
- HPC scripts → Inference: batch · SLURM templates and scripts launch inference jobs.
- HPC scripts → Generate features: batch · Cluster jobs also run feature generation at scale.
- Docs → Feature gen: documents · User guides describe feature-generation workflows.
- Docs → Training: documents · Docs explain model training and dataset regeneration.
- Docs → Inference: documents · Docs describe prediction and upload/download workflows.
- Golden data → Tests: anchors · Golden CSVs provide reference outputs for regression checks.
- Tests → PyTorch NN: verifies · Tests cover neural backend behavior.
- Tests → Rubin: verifies · Tests cover Rubin/local-parquet behavior.
- Tests → WISE: verifies · Tests cover WISE support.
- Tests → Period search: verifies · Tests cover period-search routines.