How bentian/corrdiff_input works
1. Repository purpose
- This is a data-preparation pipeline for turning heterogeneous climate model / reanalysis NetCDF inputs into CorrDiff-ready Zarr datasets.
- It is not the model itself; the core boundary is: raw atmospheric fields in NetCDF → normalized, aligned tensor fields in Zarr.
2. Main data flow
- The pipeline starts with source-specific NetCDF files from:
- ERA5 reanalysis
- TReAD high-resolution regional reanalysis
- TaiESM 3.5 km / 100 km climate model outputs
- CORDEX benchmark datasets
- These inputs are loaded, spatially aligned to a common reference grid, converted into stacked tensor channels, and written as compressed Zarr stores.
3. Primary orchestration entrypoint
- src/corrdiff_datagen.py is the main driver.
- It selects the mode based on arguments:
- CWA mode: TReAD + ERA5
- SSP mode: TaiESM 3.5 km + 100 km
- CORDEX mode: benchmark dataset generation
- This script defines the top-level pipeline boundary: load → regrid → build tensors → export.
4. Dataset assembly layer
- src/data_builder.py is the assembly layer that wires together:
- reference grid selection
- low-resolution and high-resolution dataset loading
- spatial regridding
- dataset pairing and time alignment
- This is the bridge between raw data loaders and tensor construction.
5. Source-specific loaders
- src/data_loader/ contains the source adapters:
- era5.py
- tread.py
- taiesm3p5.py
- taiesm100.py
- cordex.py
- These modules encapsulate source-specific file naming, coordinate conventions, variable selection, and any format validation needed before regridding.
6. Reference grid as the spatial backbone
- ref_grid/ contains the WRF-style coordinate grids used as the common spatial target.
- The architecture relies on a single unified reference grid per mode:
- CWA grid for TReAD/ERA5 workflows
- SSP grid for TaiESM workflows
- ref_grid/generate_wrf_coord.py makes grid generation a first-class concern, so spatial alignment is controlled outside the data sources themselves.
7. Tensor construction and metadata
- src/tensor_fields.py performs the final conversion into CorrDiff-compatible tensors.
- Its responsibilities include:
- channel stacking
- centering and scaling
- attaching metadata such as variable names, masks, and normalization parameters
- This is where the pipeline transitions from geophysical fields to model-consumable tensors.
8. Output format and storage
- The final artifact is a compressed Zarr dataset.
- Zarr is the main storage boundary because it supports chunked access and large multi-year climate datasets.
- The repo also provides utilities to inspect, filter, and merge Zarr outputs, indicating that dataset production is often done in segments and consolidated later.
9. Large-range processing strategy
- datagen_n_merge.sh exists to handle long time spans without exhausting memory.
- The workflow splits long periods into smaller chunks, generates partial Zarr stores, then merges them.
- This is an architectural workaround for scale: batch generation + merge instead of one monolithic run.
10. Helper and validation tools
- src/helpers/ contains operational tooling around the main pipeline:
- dump_zarr.py for inspection
- merge_zarr.py and filter_zarr.py for post-processing
- verify_lowres_fmt.py and verify_time_coord.py for input validation
- geo_region.py for geographic visualization
- These tools are not core data flow components, but they support debugging, sanity checks, and dataset maintenance.
11. Templates and extensibility
- src/_template/ suggests the project is designed to be extended with new data sources or loaders.
- The architecture is therefore adapter-driven: adding a new climate product should primarily mean adding a new loader and plugging it into the existing builder/tensor pipeline.
12. Operational and environment boundaries
- env/ contains Conda environment definitions, so runtime dependencies are managed as a reproducible scientific Python stack.
- .github/workflows/pylint.yml shows lightweight CI linting, but the repo’s main operational concern is still local/batch dataset generation rather than a deployed service.
- The inputs and outputs are file-based; there is no sign of a database or network service in the core architecture.
13. What to diagram
- A useful architecture diagram should show:
- multiple raw NetCDF sources feeding source-specific loaders
- loaders converging on a shared data builder
- a reference grid used for all spatial alignment
- tensor construction as the normalization/stacking stage
- Zarr as the terminal output
- helper scripts branching off for validation, debugging, filtering, and merging
- the shell script as a higher-level orchestration path for large date ranges
Main entry points: data_builder.py Builder · corrdiff_datagen.py Datagen · tensor_fields.py Tensors · datagen_n_merge.sh Batch run
Components 18
Core pipeline
Main orchestration, assembly, and tensor conversion path.
- Datagen · Main entrypoint that selects mode and drives the end-to-end dataset build.
src/corrdiff_datagen.py - Builder · Wires together loaders, reference grids, alignment, and pairing logic.
src/data_builder.py - Tensors · Converts aligned fields into CorrDiff-ready stacked tensors and metadata.
src/tensor_fields.py - Zarr store · Terminal chunked storage for CorrDiff-ready datasets.
- Batch run · Shell workflow for chunked generation and later merge over long time spans.
datagen_n_merge.sh
Data sources
Source-specific NetCDF adapters for each climate product.
- ERA5 · Loads ERA5 reanalysis fields and adapts them to the shared pipeline.
src/data_loader/era5.py - TReAD · Loads TReAD regional reanalysis inputs for CWA workflows.
src/data_loader/tread.py - TaiESM 3.5k · Loads high-resolution TaiESM climate model outputs.
src/data_loader/taiesm3p5.py - TaiESM 100k · Loads coarser TaiESM climate model outputs used alongside the 3.5 km product.
src/data_loader/taiesm100.py - CORDEX · Loads CORDEX benchmark datasets for comparison and generation workflows.
src/data_loader/cordex.py
Spatial reference
Unified reference grids and grid generation.
- Ref grid · Shared WRF-style coordinate grids used as the spatial target for alignment.
- Grid gen · Generates or regenerates WRF-style coordinate grids.
ref_grid/generate_wrf_coord.py
Operations
Inspection, validation, and merge utilities around dataset production.
- Merge Zarr · Combines partial Zarr outputs produced in long-range batch runs.
src/helpers/merge_zarr.py - Filter Zarr · Filters or subsets produced Zarr datasets during maintenance workflows.
src/helpers/filter_zarr.py - Dump Zarr · Inspects dataset contents for debugging and sanity checks.
src/helpers/dump_zarr.py - Check lowres · Validates low-resolution input formatting before data generation.
src/helpers/verify_lowres_fmt.py - Check time · Checks time-coordinate consistency across inputs and derived datasets.
src/helpers/verify_time_coord.py - Geo region · Geographic visualization helper for inspecting spatial coverage.
src/helpers/geo_region.py
Connections 19
- Datagen → Builder: builds · Entrypoint delegates dataset assembly to the builder.
- Datagen → Tensors: emits · Entrypoint reaches tensor conversion and export stage.
- Datagen → Batch run: batch mode · Long-range runs are coordinated through the shell wrapper.
- Builder → Ref grid: uses · Builder selects the common spatial target grid.
- Builder → ERA5: loads · Builder pulls ERA5 inputs through the source adapter.
- Builder → TReAD: loads · Builder pulls TReAD inputs through the source adapter.
- Builder → TaiESM 3.5k: loads · Builder pulls TaiESM 3.5 km inputs through the source adapter.
- Builder → TaiESM 100k: loads · Builder pulls TaiESM 100 km inputs through the source adapter.
- Builder → CORDEX: loads · Builder pulls CORDEX inputs through the source adapter.
- Builder → Tensors: passes aligned fields · Builder hands off regridded, time-aligned datasets for tensorization.
- Grid gen → Ref grid: creates · Grid tooling generates the WRF-style coordinate backbone.
- Tensors → Zarr store: writes · Final tensorized datasets are stored as compressed Zarr.
- Batch run → Merge Zarr: merges · Batch workflow produces partial stores and merges them later.
- Merge Zarr → Zarr store: consolidates · Merged outputs become the final dataset artifact.
- Filter Zarr → Zarr store: filters · Post-processing operates on produced Zarr stores.
- Dump Zarr → Zarr store: inspects · Inspection tooling reads Zarr outputs for debugging.
- Check lowres → Builder: guards · Validation checks protect the assembly stage from malformed inputs.
- Check time → Builder: guards · Time-coordinate checks protect pairing and alignment logic.
- Geo region → Ref grid: visualizes · Visualization helper helps inspect spatial coverage on the target grid.