How noaa-gfdl/fre-postprocess-workflow works
This repository is a Cylc workflow template for GFDL post-processing: fre-cli prepares and installs an experiment workflow, Cylc schedules its dependent tasks, and the workflow produces processed model output. Jinja filters derive task parameters and graph sections from experiment configuration; workflow applications perform analysis, time-series, and static-data work. A separate job-runner extension instruments submitted scripts for PAPIEX/EPMT operation tracking. The workflow definition and most application internals were not excerpted, so those links rely on README statements or are omitted where wiring is unclear.
Main entry points: flow.cylc Workflow definition · Jinja2Filters Jinja workflow filters · get_climatology_info.py Climatology generation · legacy_date_conversions.py Legacy date conversion
Components 19
Workflow orchestration
- Workflow definition
flow.cylc - Site task configuration
site/ppan.cylc
Configuration and graph generation
- Jinja workflow filters
Jinja2Filters - Analysis task generation
Jinja2Filters/get_analysis_info.py - Climatology generation
Jinja2Filters/get_climatology_info.py - Chunk date utilities
Jinja2Filters/iter_chunks.py - Legacy date conversion
Jinja2Filters/legacy_date_conversions.py - Remap dependencies
Jinja2Filters/form_remap_dep.py - Task parameter selection
Jinja2Filters/form_task_parameters.py - Component selection
Jinja2Filters/get_components.py
Post-processing tasks
- Time-series processing
app/make-timeseries/bin/make-timeseries - Static data combining
app/combine-statics/bin/combine-statics - Analysis execution
app/analysis/bin/analysis - Data catalog task
bin/data-catalog
Job tracking
- PPAN job handler
lib/python/ppan_handler.py - Operation tagging
lib/python/tool_ops_w_papiex.py - Tracked operations
lib/python/papiex_ops.py
Other components
- fre-cli
- Cylc engine
Connections 21
- fre-cli → Workflow definition: prepares and installs
README.md - Workflow definition → Cylc engine: runs under
README.md - Workflow definition → Jinja workflow filters: preprocesses template
README.md - Workflow definition → Site task configuration: includes site tasks
README.md - Workflow definition → Time-series processing: calls applications
README.md - Workflow definition → Static data combining: calls applications
README.md - Workflow definition → Analysis execution: calls applications
README.md - Jinja workflow filters → Analysis task generation: includes generator
README.md - Jinja workflow filters → Climatology generation: includes generator
README.md - Jinja workflow filters → Chunk date utilities: includes utility
README.md - Jinja workflow filters → Legacy date conversion: includes utility
README.md - Jinja workflow filters → Remap dependencies: includes filter
README.md - Jinja workflow filters → Task parameter selection: includes filter
README.md - Jinja workflow filters → Component selection: includes filter
README.md - Analysis task generation → Legacy date conversion: converts durations
Jinja2Filters/get_analysis_info.py - Climatology generation → Legacy date conversion: uses conversions
Jinja2Filters/get_climatology_info.py - Climatology generation → Chunk date utilities: iterates chunks
Jinja2Filters/get_climatology_info.py - Workflow definition → Data catalog task: schedules catalog tasks
- Cylc engine → PPAN job handler: submits jobs through
lib/python/ppan_handler.py - PPAN job handler → Operation tagging: tags job scripts
lib/python/ppan_handler.py - Operation tagging → Tracked operations: uses operation definitions
lib/python/tool_ops_w_papiex.py