How noaa-gfdl/spear-flp works
SPEAR-FLP provides tools for finding and transferring SPEAR climate-model datasets, preparing catalogs, and checking dataset quality. A user can search a catalog, select files, transfer them, and report QC status; the README also describes an issue-template reporting workflow. QC tools cover ensemble statistics, metadata integrity, CMOR compliance, outliers, and configuration generation. The repository includes a separate pipeline for building virtual datasets in Icechunk. Several branches are distinct utilities rather than a single orchestrated application; relationships are shown only where the README or sampled code supports them.
Main entry points: transfer_intake.py Intake transfer · transfer_data.py Catalog transfer · icechunk_rebuild.py SPEAR to Icechunk · search_catalog.ipynb Catalog search
Components 23
Catalogs and discovery
- Catalog search
examples/search_catalog.ipynb - CMIP catalog builder
utilities/cmip_catalog.py - Archive catalog builder
data_transfer/first_catalog/create_archive_catalog.py - Archive catalog merger
data_transfer/first_catalog/make_archive_decp_cats.py - Catalog filtering
data_transfer/first_catalog/spear_filter.py - File list generation
data_transfer/get_file_lists.py
Data transfer
- Intake transfer
data_transfer/transfer_intake.py - Catalog transfer
data_transfer/transfer_data.py - Rsync transfer worker
data_transfer/transfer_data.sh
Quality control
- Metadata integrity
scripts/qc/metadata_check/metadata_checker.py - QC config generator
scripts/qc/metadata_check/generate_config.py - Ensemble statistics
scripts/qc/get_ens_stats/get_ens_stats.py - Statistics engine
scripts/qc/get_ens_stats/ens_stats.py - CMOR checks
scripts/qc/check_cmor/check_cmor.py - Outlier checks
scripts/qc/outlier_checker/outlier_checker.py - QC status update
update_qc.py - QC issue form
.github/ISSUE_TEMPLATE/qc_update.yml
Cloud data
- SPEAR to Icechunk
utilities/icechunk_rebuild.py
Other components
- Data user
- QC reporter
- S3 object store
- Arraylake repository
- Slurm scheduler
Connections 15
- Data user → Catalog search: searches
README.md - Catalog search → Intake transfer: selects files
- QC reporter → QC issue form: submits report
README.md - Archive catalog builder → Slurm scheduler: runs catalog command
data_transfer/first_catalog/create_archive_catalog.py - Archive catalog merger → Archive catalog builder: builds catalogs
data_transfer/first_catalog/make_archive_decp_cats.py - Catalog filtering → File list generation: filters catalog
- Intake transfer → Slurm scheduler: submits jobs
data_transfer/transfer_intake.py - Intake transfer → Rsync transfer worker: invokes worker
data_transfer/transfer_intake.py - Catalog transfer → Slurm scheduler: submits jobs
data_transfer/transfer_data.py - Catalog transfer → Rsync transfer worker: invokes worker
data_transfer/transfer_data.py - Ensemble statistics → Statistics engine: calls statistics
scripts/qc/get_ens_stats/get_ens_stats.py - QC config generator → Metadata integrity: configures checks
- QC issue form → QC status update: records status
- SPEAR to Icechunk → S3 object store: reads datasets
utilities/icechunk_rebuild.py - SPEAR to Icechunk → Arraylake repository: creates repository
utilities/icechunk_rebuild.py