# noaa-gfdl/spear-flp architecture

> Architecture diagram of the GitHub repository noaa-gfdl/spear-flp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noaa-gfdl/spear-flp
- Repository: https://github.com/noaa-gfdl/spear-flp

## Overview

SPEAR-FLP provides tools for finding and transferring SPEAR climate-model datasets, preparing catalogs, and checking dataset quality. A user can search a catalog, select files, transfer them, and report QC status; the README also describes an issue-template reporting workflow. QC tools cover ensemble statistics, metadata integrity, CMOR compliance, outliers, and configuration generation. The repository includes a separate pipeline for building virtual datasets in Icechunk. Several branches are distinct utilities rather than a single orchestrated application; relationships are shown only where the README or sampled code supports them.

Main entry points: [`data_transfer/transfer_intake.py`](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_intake.py) (Intake transfer), [`data_transfer/transfer_data.py`](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_data.py) (Catalog transfer), [`utilities/icechunk_rebuild.py`](https://github.com/noaa-gfdl/spear-flp/blob/main/utilities/icechunk_rebuild.py) (SPEAR to Icechunk), [`examples/search_catalog.ipynb`](https://github.com/noaa-gfdl/spear-flp/blob/main/examples/search_catalog.ipynb) (Catalog search)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_catalogs["Catalogs and discovery"]
  node_catalog_search["Catalog search"]
  node_cmip_catalog["CMIP catalog builder<br/>[cmip_catalog.py]"]
  node_archive_catalog["Archive catalog builder"]
  node_archive_catalog_merge["Archive catalog merger"]
  node_catalog_filter["Catalog filtering<br/>[spear_filter.py]"]
  node_file_lists["File list generation<br/>[get_file_lists.py]"]
end

subgraph group_transfer["Data transfer"]
  node_transfer_intake["Intake transfer<br/>[transfer_intake.py]"]
  node_transfer_data["Catalog transfer<br/>[transfer_data.py]"]
  node_transfer_script["Rsync transfer worker<br/>[transfer_data.sh]"]
end

subgraph group_qc["Quality control"]
  node_metadata_qc["Metadata integrity"]
  node_config_generator["QC config generator<br/>[generate_config.py]"]
  node_ensemble_stats["Ensemble statistics<br/>[get_ens_stats.py]"]
  node_stats_engine["Statistics engine<br/>[ens_stats.py]"]
  node_cmor_check["CMOR checks<br/>[check_cmor.py]"]
  node_outlier_check["Outlier checks<br/>[outlier_checker.py]"]
  node_qc_update["QC status update<br/>[update_qc.py]"]
  node_issue_template["QC issue form<br/>[qc_update.yml]"]
end

subgraph group_cloud["Cloud data"]
  node_icechunk_builder["SPEAR to Icechunk"]
end

node_user(("Data user"))
node_reporter(("QC reporter"))
node_s3[("S3 object store")]
node_arraylake[("Arraylake repository")]
node_scheduler["Slurm scheduler"]

node_user -->|"searches"| node_catalog_search
node_catalog_search -.->|"selects files"| node_transfer_intake
node_reporter -->|"submits report"| node_issue_template
node_archive_catalog -->|"runs catalog command"| node_scheduler
node_archive_catalog_merge -->|"builds catalogs"| node_archive_catalog
node_catalog_filter -.->|"filters catalog"| node_file_lists
node_transfer_intake -->|"submits jobs"| node_scheduler
node_transfer_intake -->|"invokes worker"| node_transfer_script
node_transfer_data -->|"submits jobs"| node_scheduler
node_transfer_data -->|"invokes worker"| node_transfer_script
node_ensemble_stats -->|"calls statistics"| node_stats_engine
node_config_generator -.->|"configures checks"| node_metadata_qc
node_issue_template -.->|"records status"| node_qc_update
node_icechunk_builder -->|"reads datasets"| node_s3
node_icechunk_builder -->|"creates repository"| node_arraylake

click node_catalog_search "https://github.com/noaa-gfdl/spear-flp/blob/main/examples/search_catalog.ipynb"
click node_cmip_catalog "https://github.com/noaa-gfdl/spear-flp/blob/main/utilities/cmip_catalog.py"
click node_archive_catalog "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/first_catalog/create_archive_catalog.py"
click node_archive_catalog_merge "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/first_catalog/make_archive_decp_cats.py"
click node_catalog_filter "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/first_catalog/spear_filter.py"
click node_file_lists "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/get_file_lists.py"
click node_transfer_intake "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_intake.py"
click node_transfer_data "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_data.py"
click node_transfer_script "https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_data.sh"
click node_metadata_qc "https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/metadata_check/metadata_checker.py"
click node_config_generator "https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/metadata_check/generate_config.py"
click node_ensemble_stats "https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/get_ens_stats/get_ens_stats.py"
click node_stats_engine "https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/get_ens_stats/ens_stats.py"
click node_cmor_check "https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/check_cmor/check_cmor.py"
click node_outlier_check "https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/outlier_checker/outlier_checker.py"
click node_qc_update "https://github.com/noaa-gfdl/spear-flp/blob/main/update_qc.py"
click node_issue_template "https://github.com/noaa-gfdl/spear-flp/blob/main/.github/ISSUE_TEMPLATE/qc_update.yml"
click node_icechunk_builder "https://github.com/noaa-gfdl/spear-flp/blob/main/utilities/icechunk_rebuild.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_catalog_search,node_cmip_catalog,node_archive_catalog,node_archive_catalog_merge,node_catalog_filter,node_file_lists,node_user toneBlue
class node_transfer_intake,node_transfer_data,node_transfer_script,node_s3,node_arraylake toneAmber
class node_metadata_qc,node_config_generator,node_ensemble_stats,node_stats_engine,node_cmor_check,node_outlier_check,node_qc_update,node_issue_template toneMint
class node_icechunk_builder,node_scheduler toneRose
class node_reporter toneIndigo
```

## Components

### Catalogs and discovery

- [Catalog search](https://github.com/noaa-gfdl/spear-flp/blob/main/examples/search_catalog.ipynb): `examples/search_catalog.ipynb`
- [CMIP catalog builder](https://github.com/noaa-gfdl/spear-flp/blob/main/utilities/cmip_catalog.py): `utilities/cmip_catalog.py`
- [Archive catalog builder](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/first_catalog/create_archive_catalog.py): `data_transfer/first_catalog/create_archive_catalog.py`
- [Archive catalog merger](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/first_catalog/make_archive_decp_cats.py): `data_transfer/first_catalog/make_archive_decp_cats.py`
- [Catalog filtering](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/first_catalog/spear_filter.py): `data_transfer/first_catalog/spear_filter.py`
- [File list generation](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/get_file_lists.py): `data_transfer/get_file_lists.py`

### Data transfer

- [Intake transfer](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_intake.py): `data_transfer/transfer_intake.py`
- [Catalog transfer](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_data.py): `data_transfer/transfer_data.py`
- [Rsync transfer worker](https://github.com/noaa-gfdl/spear-flp/blob/main/data_transfer/transfer_data.sh): `data_transfer/transfer_data.sh`

### Quality control

- [Metadata integrity](https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/metadata_check/metadata_checker.py): `scripts/qc/metadata_check/metadata_checker.py`
- [QC config generator](https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/metadata_check/generate_config.py): `scripts/qc/metadata_check/generate_config.py`
- [Ensemble statistics](https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/get_ens_stats/get_ens_stats.py): `scripts/qc/get_ens_stats/get_ens_stats.py`
- [Statistics engine](https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/get_ens_stats/ens_stats.py): `scripts/qc/get_ens_stats/ens_stats.py`
- [CMOR checks](https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/check_cmor/check_cmor.py): `scripts/qc/check_cmor/check_cmor.py`
- [Outlier checks](https://github.com/noaa-gfdl/spear-flp/blob/main/scripts/qc/outlier_checker/outlier_checker.py): `scripts/qc/outlier_checker/outlier_checker.py`
- [QC status update](https://github.com/noaa-gfdl/spear-flp/blob/main/update_qc.py): `update_qc.py`
- [QC issue form](https://github.com/noaa-gfdl/spear-flp/blob/main/.github/ISSUE_TEMPLATE/qc_update.yml): `.github/ISSUE_TEMPLATE/qc_update.yml`

### Cloud data

- [SPEAR to Icechunk](https://github.com/noaa-gfdl/spear-flp/blob/main/utilities/icechunk_rebuild.py): `utilities/icechunk_rebuild.py`

### Other components

- **Data user**
- **QC reporter**
- **S3 object store**
- **Arraylake repository**
- **Slurm scheduler**

## Connections

- Data user → Catalog search: searches (evidence: `README.md`)
- Catalog search → Intake transfer: selects files
- QC reporter → QC issue form: submits report (evidence: `README.md`)
- Archive catalog builder → Slurm scheduler: runs catalog command (evidence: `data_transfer/first_catalog/create_archive_catalog.py`)
- Archive catalog merger → Archive catalog builder: builds catalogs (evidence: `data_transfer/first_catalog/make_archive_decp_cats.py`)
- Catalog filtering → File list generation: filters catalog
- Intake transfer → Slurm scheduler: submits jobs (evidence: `data_transfer/transfer_intake.py`)
- Intake transfer → Rsync transfer worker: invokes worker (evidence: `data_transfer/transfer_intake.py`)
- Catalog transfer → Slurm scheduler: submits jobs (evidence: `data_transfer/transfer_data.py`)
- Catalog transfer → Rsync transfer worker: invokes worker (evidence: `data_transfer/transfer_data.py`)
- Ensemble statistics → Statistics engine: calls statistics (evidence: `scripts/qc/get_ens_stats/get_ens_stats.py`)
- QC config generator → Metadata integrity: configures checks
- QC issue form → QC status update: records status
- SPEAR to Icechunk → S3 object store: reads datasets (evidence: `utilities/icechunk_rebuild.py`)
- SPEAR to Icechunk → Arraylake repository: creates repository (evidence: `utilities/icechunk_rebuild.py`)