# bentian/corrdiff_input architecture

> Architecture diagram of the GitHub repository bentian/corrdiff_input, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bentian/corrdiff_input
- Repository: https://github.com/bentian/corrdiff_input

## Overview

1. **Repository purpose**
   - This is a **data-preparation pipeline** for turning heterogeneous climate model / reanalysis NetCDF inputs into **CorrDiff-ready Zarr datasets**.
   - It is not the model itself; the core boundary is: **raw atmospheric fields in NetCDF → normalized, aligned tensor fields in Zarr**.

2. **Main data flow**
   - The pipeline starts with source-specific NetCDF files from:
     - **ERA5** reanalysis
     - **TReAD** high-resolution regional reanalysis
     - **TaiESM 3.5 km / 100 km** climate model outputs
     - **CORDEX** benchmark datasets
   - These inputs are loaded, spatially aligned to a common reference grid, converted into stacked tensor channels, and written as compressed **Zarr stores**.

3. **Primary orchestration entrypoint**
   - `src/corrdiff_datagen.py` is the main driver.
   - It selects the mode based on arguments:
     - CWA mode: **TReAD + ERA5**
     - SSP mode: **TaiESM 3.5 km + 100 km**
     - CORDEX mode: benchmark dataset generation
   - This script defines the top-level pipeline boundary: load → regrid → build tensors → export.

4. **Dataset assembly layer**
   - `src/data_builder.py` is the assembly layer that wires together:
     - reference grid selection
     - low-resolution and high-resolution dataset loading
     - spatial regridding
     - dataset pairing and time alignment
   - This is the bridge between raw data loaders and tensor construction.

5. **Source-specific loaders**
   - `src/data_loader/` contains the source adapters:
     - `era5.py`
     - `tread.py`
     - `taiesm3p5.py`
     - `taiesm100.py`
     - `cordex.py`
   - These modules encapsulate source-specific file naming, coordinate conventions, variable selection, and any format validation needed before regridding.

6. **Reference grid as the spatial backbone**
   - `ref_grid/` contains the WRF-style coordinate grids used as the common spatial target.
   - The architecture relies on a **single unified reference grid** per mode:
     - CWA grid for TReAD/ERA5 workflows
     - SSP grid for TaiESM workflows
   - `ref_grid/generate_wrf_coord.py` makes grid generation a first-class concern, so spatial alignment is controlled outside the data sources themselves.

7. **Tensor construction and metadata**
   - `src/tensor_fields.py` performs the final conversion into CorrDiff-compatible tensors.
   - Its responsibilities include:
     - channel stacking
     - centering and scaling
     - attaching metadata such as variable names, masks, and normalization parameters
   - This is where the pipeline transitions from geophysical fields to model-consumable tensors.

8. **Output format and storage**
   - The final artifact is a **compressed Zarr dataset**.
   - Zarr is the main storage boundary because it supports chunked access and large multi-year climate datasets.
   - The repo also provides utilities to inspect, filter, and merge Zarr outputs, indicating that dataset production is often done in segments and consolidated later.

9. **Large-range processing strategy**
   - `datagen_n_merge.sh` exists to handle long time spans without exhausting memory.
   - The workflow splits long periods into smaller chunks, generates partial Zarr stores, then merges them.
   - This is an architectural workaround for scale: **batch generation + merge** instead of one monolithic run.

10. **Helper and validation tools**
    - `src/helpers/` contains operational tooling around the main pipeline:
      - `dump_zarr.py` for inspection
      - `merge_zarr.py` and `filter_zarr.py` for post-processing
      - `verify_lowres_fmt.py` and `verify_time_coord.py` for input validation
      - `geo_region.py` for geographic visualization
    - These tools are not core data flow components, but they support debugging, sanity checks, and dataset maintenance.

11. **Templates and extensibility**
    - `src/_template/` suggests the project is designed to be extended with new data sources or loaders.
    - The architecture is therefore **adapter-driven**: adding a new climate product should primarily mean adding a new loader and plugging it into the existing builder/tensor pipeline.

12. **Operational and environment boundaries**
    - `env/` contains Conda environment definitions, so runtime dependencies are managed as a reproducible scientific Python stack.
    - `.github/workflows/pylint.yml` shows lightweight CI linting, but the repo’s main operational concern is still local/batch dataset generation rather than a deployed service.
    - The inputs and outputs are file-based; there is no sign of a database or network service in the core architecture.

13. **What to diagram**
    - A useful architecture diagram should show:
      - multiple raw NetCDF sources feeding source-specific loaders
      - loaders converging on a shared data builder
      - a reference grid used for all spatial alignment
      - tensor construction as the normalization/stacking stage
      - Zarr as the terminal output
      - helper scripts branching off for validation, debugging, filtering, and merging
      - the shell script as a higher-level orchestration path for large date ranges

Main entry points: [`src/data_builder.py`](https://github.com/bentian/corrdiff_input/blob/main/src/data_builder.py) (Builder), [`src/corrdiff_datagen.py`](https://github.com/bentian/corrdiff_input/blob/main/src/corrdiff_datagen.py) (Datagen), [`src/tensor_fields.py`](https://github.com/bentian/corrdiff_input/blob/main/src/tensor_fields.py) (Tensors), [`datagen_n_merge.sh`](https://github.com/bentian/corrdiff_input/blob/main/datagen_n_merge.sh) (Batch run)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Core pipeline"]
  node_corrdiff_datagen["Datagen<br/>orchestrator"]
  node_data_builder["Builder<br/>assembly<br/>[data_builder.py]"]
  node_tensor_fields["Tensors<br/>tensorization<br/>[tensor_fields.py]"]
  node_zarr_store[("Zarr store<br/>output format")]
  node_batch_script["Batch run<br/>shell orchestration<br/>[datagen_n_merge.sh]"]
end

subgraph group_sources["Data sources"]
  node_era5_loader["ERA5<br/>netcdf adapter<br/>[era5.py]"]
  node_tread_loader["TReAD<br/>netcdf adapter<br/>[tread.py]"]
  node_taiesm3p5_loader["TaiESM 3.5k<br/>netcdf adapter<br/>[taiesm3p5.py]"]
  node_taiesm100_loader["TaiESM 100k<br/>netcdf adapter<br/>[taiesm100.py]"]
  node_cordex_loader["CORDEX<br/>netcdf adapter<br/>[cordex.py]"]
end

subgraph group_spatial["Spatial reference"]
  node_ref_grid[("Ref grid<br/>spatial backbone")]
  node_grid_gen["Grid gen<br/>grid tooling"]
end

subgraph group_ops["Operations"]
  node_merge_zarr["Merge Zarr<br/>postprocess<br/>[merge_zarr.py]"]
  node_filter_zarr["Filter Zarr<br/>postprocess<br/>[filter_zarr.py]"]
  node_dump_zarr["Dump Zarr<br/>inspect<br/>[dump_zarr.py]"]
  node_verify_lowres["Check lowres<br/>validation"]
  node_verify_time["Check time<br/>validation"]
  node_geo_region(("Geo region<br/>viz helper<br/>[geo_region.py]"))
end

node_corrdiff_datagen -->|"builds"| node_data_builder
node_corrdiff_datagen -->|"emits"| node_tensor_fields
node_corrdiff_datagen -.->|"batch mode"| node_batch_script
node_data_builder -->|"uses"| node_ref_grid
node_data_builder -->|"loads"| node_era5_loader
node_data_builder -->|"loads"| node_tread_loader
node_data_builder -->|"loads"| node_taiesm3p5_loader
node_data_builder -->|"loads"| node_taiesm100_loader
node_data_builder -->|"loads"| node_cordex_loader
node_data_builder -->|"passes aligned fields"| node_tensor_fields
node_grid_gen -.->|"creates"| node_ref_grid
node_tensor_fields -->|"writes"| node_zarr_store
node_batch_script -->|"merges"| node_merge_zarr
node_merge_zarr -->|"consolidates"| node_zarr_store
node_filter_zarr -.->|"filters"| node_zarr_store
node_dump_zarr -.->|"inspects"| node_zarr_store
node_verify_lowres -.->|"guards"| node_data_builder
node_verify_time -.->|"guards"| node_data_builder
node_geo_region -.->|"visualizes"| node_ref_grid

click node_corrdiff_datagen "https://github.com/bentian/corrdiff_input/blob/main/src/corrdiff_datagen.py"
click node_data_builder "https://github.com/bentian/corrdiff_input/blob/main/src/data_builder.py"
click node_tensor_fields "https://github.com/bentian/corrdiff_input/blob/main/src/tensor_fields.py"
click node_era5_loader "https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/era5.py"
click node_tread_loader "https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/tread.py"
click node_taiesm3p5_loader "https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/taiesm3p5.py"
click node_taiesm100_loader "https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/taiesm100.py"
click node_cordex_loader "https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/cordex.py"
click node_grid_gen "https://github.com/bentian/corrdiff_input/blob/main/ref_grid/generate_wrf_coord.py"
click node_merge_zarr "https://github.com/bentian/corrdiff_input/blob/main/src/helpers/merge_zarr.py"
click node_filter_zarr "https://github.com/bentian/corrdiff_input/blob/main/src/helpers/filter_zarr.py"
click node_dump_zarr "https://github.com/bentian/corrdiff_input/blob/main/src/helpers/dump_zarr.py"
click node_verify_lowres "https://github.com/bentian/corrdiff_input/blob/main/src/helpers/verify_lowres_fmt.py"
click node_verify_time "https://github.com/bentian/corrdiff_input/blob/main/src/helpers/verify_time_coord.py"
click node_geo_region "https://github.com/bentian/corrdiff_input/blob/main/src/helpers/geo_region.py"
click node_batch_script "https://github.com/bentian/corrdiff_input/blob/main/datagen_n_merge.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_corrdiff_datagen,node_data_builder,node_tensor_fields,node_zarr_store,node_batch_script toneBlue
class node_era5_loader,node_tread_loader,node_taiesm3p5_loader,node_taiesm100_loader,node_cordex_loader toneAmber
class node_ref_grid,node_grid_gen toneMint
class node_merge_zarr,node_filter_zarr,node_dump_zarr,node_verify_lowres,node_verify_time,node_geo_region toneRose
```

## Components

### Core pipeline

Main orchestration, assembly, and tensor conversion path.

- [Datagen](https://github.com/bentian/corrdiff_input/blob/main/src/corrdiff_datagen.py): orchestrator · Main entrypoint that selects mode and drives the end-to-end dataset build. · `src/corrdiff_datagen.py`
- [Builder](https://github.com/bentian/corrdiff_input/blob/main/src/data_builder.py): assembly · Wires together loaders, reference grids, alignment, and pairing logic. · `src/data_builder.py`
- [Tensors](https://github.com/bentian/corrdiff_input/blob/main/src/tensor_fields.py): tensorization · Converts aligned fields into CorrDiff-ready stacked tensors and metadata. · `src/tensor_fields.py`
- **Zarr store**: output format · Terminal chunked storage for CorrDiff-ready datasets.
- [Batch run](https://github.com/bentian/corrdiff_input/blob/main/datagen_n_merge.sh): shell orchestration · Shell workflow for chunked generation and later merge over long time spans. · `datagen_n_merge.sh`

### Data sources

Source-specific NetCDF adapters for each climate product.

- [ERA5](https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/era5.py): netcdf adapter · Loads ERA5 reanalysis fields and adapts them to the shared pipeline. · `src/data_loader/era5.py`
- [TReAD](https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/tread.py): netcdf adapter · Loads TReAD regional reanalysis inputs for CWA workflows. · `src/data_loader/tread.py`
- [TaiESM 3.5k](https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/taiesm3p5.py): netcdf adapter · Loads high-resolution TaiESM climate model outputs. · `src/data_loader/taiesm3p5.py`
- [TaiESM 100k](https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/taiesm100.py): netcdf adapter · Loads coarser TaiESM climate model outputs used alongside the 3.5 km product. · `src/data_loader/taiesm100.py`
- [CORDEX](https://github.com/bentian/corrdiff_input/blob/main/src/data_loader/cordex.py): netcdf adapter · Loads CORDEX benchmark datasets for comparison and generation workflows. · `src/data_loader/cordex.py`

### Spatial reference

Unified reference grids and grid generation.

- **Ref grid**: spatial backbone · Shared WRF-style coordinate grids used as the spatial target for alignment.
- [Grid gen](https://github.com/bentian/corrdiff_input/blob/main/ref_grid/generate_wrf_coord.py): grid tooling · Generates or regenerates WRF-style coordinate grids. · `ref_grid/generate_wrf_coord.py`

### Operations

Inspection, validation, and merge utilities around dataset production.

- [Merge Zarr](https://github.com/bentian/corrdiff_input/blob/main/src/helpers/merge_zarr.py): postprocess · Combines partial Zarr outputs produced in long-range batch runs. · `src/helpers/merge_zarr.py`
- [Filter Zarr](https://github.com/bentian/corrdiff_input/blob/main/src/helpers/filter_zarr.py): postprocess · Filters or subsets produced Zarr datasets during maintenance workflows. · `src/helpers/filter_zarr.py`
- [Dump Zarr](https://github.com/bentian/corrdiff_input/blob/main/src/helpers/dump_zarr.py): inspect · Inspects dataset contents for debugging and sanity checks. · `src/helpers/dump_zarr.py`
- [Check lowres](https://github.com/bentian/corrdiff_input/blob/main/src/helpers/verify_lowres_fmt.py): validation · Validates low-resolution input formatting before data generation. · `src/helpers/verify_lowres_fmt.py`
- [Check time](https://github.com/bentian/corrdiff_input/blob/main/src/helpers/verify_time_coord.py): validation · Checks time-coordinate consistency across inputs and derived datasets. · `src/helpers/verify_time_coord.py`
- [Geo region](https://github.com/bentian/corrdiff_input/blob/main/src/helpers/geo_region.py): viz helper · Geographic visualization helper for inspecting spatial coverage. · `src/helpers/geo_region.py`

## Connections

- Datagen → Builder: builds · Entrypoint delegates dataset assembly to the builder.
- Datagen → Tensors: emits · Entrypoint reaches tensor conversion and export stage.
- Datagen → Batch run: batch mode · Long-range runs are coordinated through the shell wrapper.
- Builder → Ref grid: uses · Builder selects the common spatial target grid.
- Builder → ERA5: loads · Builder pulls ERA5 inputs through the source adapter.
- Builder → TReAD: loads · Builder pulls TReAD inputs through the source adapter.
- Builder → TaiESM 3.5k: loads · Builder pulls TaiESM 3.5 km inputs through the source adapter.
- Builder → TaiESM 100k: loads · Builder pulls TaiESM 100 km inputs through the source adapter.
- Builder → CORDEX: loads · Builder pulls CORDEX inputs through the source adapter.
- Builder → Tensors: passes aligned fields · Builder hands off regridded, time-aligned datasets for tensorization.
- Grid gen → Ref grid: creates · Grid tooling generates the WRF-style coordinate backbone.
- Tensors → Zarr store: writes · Final tensorized datasets are stored as compressed Zarr.
- Batch run → Merge Zarr: merges · Batch workflow produces partial stores and merges them later.
- Merge Zarr → Zarr store: consolidates · Merged outputs become the final dataset artifact.
- Filter Zarr → Zarr store: filters · Post-processing operates on produced Zarr stores.
- Dump Zarr → Zarr store: inspects · Inspection tooling reads Zarr outputs for debugging.
- Check lowres → Builder: guards · Validation checks protect the assembly stage from malformed inputs.
- Check time → Builder: guards · Time-coordinate checks protect pairing and alignment logic.
- Geo region → Ref grid: visualizes · Visualization helper helps inspect spatial coverage on the target grid.
