# noaa-gfdl/fre-postprocess-workflow architecture

> Architecture diagram of the GitHub repository noaa-gfdl/fre-postprocess-workflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noaa-gfdl/fre-postprocess-workflow
- Repository: https://github.com/noaa-gfdl/fre-postprocess-workflow

## Overview

This repository is a Cylc workflow template for GFDL post-processing: fre-cli prepares and installs an experiment workflow, Cylc schedules its dependent tasks, and the workflow produces processed model output. Jinja filters derive task parameters and graph sections from experiment configuration; workflow applications perform analysis, time-series, and static-data work. A separate job-runner extension instruments submitted scripts for PAPIEX/EPMT operation tracking. The workflow definition and most application internals were not excerpted, so those links rely on README statements or are omitted where wiring is unclear.

Main entry points: [`flow.cylc`](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/flow.cylc) (Workflow definition), [`Jinja2Filters`](https://github.com/noaa-gfdl/fre-postprocess-workflow/tree/main/Jinja2Filters) (Jinja workflow filters), [`Jinja2Filters/get_climatology_info.py`](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_climatology_info.py) (Climatology generation), [`Jinja2Filters/legacy_date_conversions.py`](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/legacy_date_conversions.py) (Legacy date conversion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Workflow orchestration"]
  node_workflow["Workflow definition<br/>[flow.cylc]"]
  node_site_tasks["Site task configuration<br/>[ppan.cylc]"]
end

subgraph group_config["Configuration and graph generation"]
  node_filters["Jinja workflow filters"]
  node_analysis_config["Analysis task generation"]
  node_climo_config["Climatology generation"]
  node_chunking["Chunk date utilities<br/>[iter_chunks.py]"]
  node_date_conversion["Legacy date conversion"]
  node_remap_dependencies["Remap dependencies<br/>[form_remap_dep.py]"]
  node_task_parameters["Task parameter selection"]
  node_component_selection["Component selection<br/>[get_components.py]"]
end

subgraph group_processing["Post-processing tasks"]
  node_timeseries["Time-series processing"]
  node_combine_statics["Static data combining"]
  node_analysis["Analysis execution"]
  node_data_catalog["Data catalog task"]
end

subgraph group_tracking["Job tracking"]
  node_job_handler["PPAN job handler<br/>[ppan_handler.py]"]
  node_operation_tagger["Operation tagging"]
  node_operation_catalog["Tracked operations<br/>[papiex_ops.py]"]
end

node_frecli(("fre-cli"))
node_cylc["Cylc engine"]

node_frecli -->|"prepares and installs"| node_workflow
node_workflow -->|"runs under"| node_cylc
node_workflow -->|"preprocesses template"| node_filters
node_workflow -->|"includes site tasks"| node_site_tasks
node_workflow -->|"calls applications"| node_timeseries
node_workflow -->|"calls applications"| node_combine_statics
node_workflow -->|"calls applications"| node_analysis
node_filters -->|"includes generator"| node_analysis_config
node_filters -->|"includes generator"| node_climo_config
node_filters -->|"includes utility"| node_chunking
node_filters -->|"includes utility"| node_date_conversion
node_filters -->|"includes filter"| node_remap_dependencies
node_filters -->|"includes filter"| node_task_parameters
node_filters -->|"includes filter"| node_component_selection
node_analysis_config -->|"converts durations"| node_date_conversion
node_climo_config -->|"uses conversions"| node_date_conversion
node_climo_config -->|"iterates chunks"| node_chunking
node_workflow -->|"schedules catalog tasks"| node_data_catalog
node_cylc -->|"submits jobs through"| node_job_handler
node_job_handler -->|"tags job scripts"| node_operation_tagger
node_operation_tagger -->|"uses operation definitions"| node_operation_catalog

click node_workflow "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/flow.cylc"
click node_site_tasks "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/site/ppan.cylc"
click node_filters "https://github.com/noaa-gfdl/fre-postprocess-workflow/tree/main/Jinja2Filters"
click node_analysis_config "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_analysis_info.py"
click node_climo_config "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_climatology_info.py"
click node_chunking "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/iter_chunks.py"
click node_date_conversion "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/legacy_date_conversions.py"
click node_remap_dependencies "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/form_remap_dep.py"
click node_task_parameters "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/form_task_parameters.py"
click node_component_selection "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_components.py"
click node_timeseries "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/app/make-timeseries/bin/make-timeseries"
click node_combine_statics "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/app/combine-statics/bin/combine-statics"
click node_analysis "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/app/analysis/bin/analysis"
click node_data_catalog "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/bin/data-catalog"
click node_job_handler "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/lib/python/ppan_handler.py"
click node_operation_tagger "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/lib/python/tool_ops_w_papiex.py"
click node_operation_catalog "https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/lib/python/papiex_ops.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workflow,node_site_tasks toneBlue
class node_filters,node_analysis_config,node_climo_config,node_chunking,node_date_conversion,node_remap_dependencies,node_task_parameters,node_component_selection toneAmber
class node_timeseries,node_combine_statics,node_analysis,node_data_catalog toneMint
class node_job_handler,node_operation_tagger,node_operation_catalog toneRose
class node_frecli,node_cylc toneIndigo
```

## Components

### Workflow orchestration

- [Workflow definition](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/flow.cylc): `flow.cylc`
- [Site task configuration](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/site/ppan.cylc): `site/ppan.cylc`

### Configuration and graph generation

- [Jinja workflow filters](https://github.com/noaa-gfdl/fre-postprocess-workflow/tree/main/Jinja2Filters): `Jinja2Filters`
- [Analysis task generation](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_analysis_info.py): `Jinja2Filters/get_analysis_info.py`
- [Climatology generation](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_climatology_info.py): `Jinja2Filters/get_climatology_info.py`
- [Chunk date utilities](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/iter_chunks.py): `Jinja2Filters/iter_chunks.py`
- [Legacy date conversion](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/legacy_date_conversions.py): `Jinja2Filters/legacy_date_conversions.py`
- [Remap dependencies](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/form_remap_dep.py): `Jinja2Filters/form_remap_dep.py`
- [Task parameter selection](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/form_task_parameters.py): `Jinja2Filters/form_task_parameters.py`
- [Component selection](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/Jinja2Filters/get_components.py): `Jinja2Filters/get_components.py`

### Post-processing tasks

- [Time-series processing](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/app/make-timeseries/bin/make-timeseries): `app/make-timeseries/bin/make-timeseries`
- [Static data combining](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/app/combine-statics/bin/combine-statics): `app/combine-statics/bin/combine-statics`
- [Analysis execution](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/app/analysis/bin/analysis): `app/analysis/bin/analysis`
- [Data catalog task](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/bin/data-catalog): `bin/data-catalog`

### Job tracking

- [PPAN job handler](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/lib/python/ppan_handler.py): `lib/python/ppan_handler.py`
- [Operation tagging](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/lib/python/tool_ops_w_papiex.py): `lib/python/tool_ops_w_papiex.py`
- [Tracked operations](https://github.com/noaa-gfdl/fre-postprocess-workflow/blob/main/lib/python/papiex_ops.py): `lib/python/papiex_ops.py`

### Other components

- **fre-cli**
- **Cylc engine**

## Connections

- fre-cli → Workflow definition: prepares and installs (evidence: `README.md`)
- Workflow definition → Cylc engine: runs under (evidence: `README.md`)
- Workflow definition → Jinja workflow filters: preprocesses template (evidence: `README.md`)
- Workflow definition → Site task configuration: includes site tasks (evidence: `README.md`)
- Workflow definition → Time-series processing: calls applications (evidence: `README.md`)
- Workflow definition → Static data combining: calls applications (evidence: `README.md`)
- Workflow definition → Analysis execution: calls applications (evidence: `README.md`)
- Jinja workflow filters → Analysis task generation: includes generator (evidence: `README.md`)
- Jinja workflow filters → Climatology generation: includes generator (evidence: `README.md`)
- Jinja workflow filters → Chunk date utilities: includes utility (evidence: `README.md`)
- Jinja workflow filters → Legacy date conversion: includes utility (evidence: `README.md`)
- Jinja workflow filters → Remap dependencies: includes filter (evidence: `README.md`)
- Jinja workflow filters → Task parameter selection: includes filter (evidence: `README.md`)
- Jinja workflow filters → Component selection: includes filter (evidence: `README.md`)
- Analysis task generation → Legacy date conversion: converts durations (evidence: `Jinja2Filters/get_analysis_info.py`)
- Climatology generation → Legacy date conversion: uses conversions (evidence: `Jinja2Filters/get_climatology_info.py`)
- Climatology generation → Chunk date utilities: iterates chunks (evidence: `Jinja2Filters/get_climatology_info.py`)
- Workflow definition → Data catalog task: schedules catalog tasks
- Cylc engine → PPAN job handler: submits jobs through (evidence: `lib/python/ppan_handler.py`)
- PPAN job handler → Operation tagging: tags job scripts (evidence: `lib/python/ppan_handler.py`)
- Operation tagging → Tracked operations: uses operation definitions (evidence: `lib/python/tool_ops_w_papiex.py`)