# john-amal/ifs_ic_gen architecture

> Architecture diagram of the GitHub repository john-amal/ifs_ic_gen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/john-amal/ifs_ic_gen
- Repository: https://github.com/john-amal/ifs_ic_gen

## Overview

##### Project type & purpose (what you’re diagramming)
This repo is not an application/service architecture; it’s an **HPC batch-style data processing utility**: a single Bash pipeline that **generates IFS atmosphere initial condition (IC) files** for branching an **IFS–FESOM coupled model** run.

Key points from README + tree:
- Core artifact: `generate_initial_conditions_parallel_only_IFS.sh` (the “system”).
- Inputs: **IFS restart GRIB files** produced by a special IFS build (“IC Gen build”) from a short run.
- Processing: split/modify GRIB, per-level parallelization, optional interpolation (Gaussian→spectral).
- Outputs: **three IFS IC GRIB files** (INIT/INIUA) + temp working directories.
- External dependencies: `grib_filter`, `grib_set` (ecCodes), `parallel`, optional `cdo` (<=2.4 for `-C`).

- `generate_initial_conditions_parallel_only_IFS.sh`: **Orchestrator / pipeline**

- **runtime environment** (HPC modules, filesystem paths)
- **input/output GRIB artifacts**
- **processing stages inside the script**
- **external tools** called by the script

##### Field modifications / metadata edits (`grib_set`)
5. **Per-level parallel processing** (GNU `parallel`, especially for 137 levels for upper-air)
##### Optional interpolation (`cdo gp2sp` etc. when `-C`)
- interpolate `lnsp`, `vo`, `d`, `t` from reduced Gaussian to spectral
7. **Assembly/merge into final IC products** (concatenate/merge GRIB messages into 3 output files)

##### Outputs (data store)
In `<outdir>/IC_<idate>/`:
- `ICMSH<expid>INIT` (spectral core fields)
- `ICMGG<expid>INIT` (surface grid-point fields)
- `ICMGG<expid>INIUA` (upper-air grid-point fields, all 137 levels)

Plus temps (explicitly show they are ephemeral):
- `<outdir>/tmpdir/`
- `<outdir>/ictmpdir/`

##### Downstream consumer (optional but useful)
- **Next IFS-FESOM branched run** consumes these three IC files.
Even though “FESOM ICs handled separately,” call that out as a boundary: this pipeline outputs **IFS-atmos-only** ICs.

##### Architectural patterns / design principles to highlight
This project’s “architecture” is basically:
- **Batch ETL pipeline** (Extract GRIB → Transform fields/metadata/interp → Load assembled GRIB)
- **Embarrassingly parallel level processing** (137 levels), implemented via GNU `parallel`
- **Tool-based composition** (thin orchestration script around stable CLI utilities)
- **Strong file-contract interfaces** (names/patterns like `ICMGG<expid>+<step>`)

- Restart file naming conventions and presence in `<expdir>`
- Exact tool availability/versions (esp. `cdo` constraint)

##### “Don’t overthink it” detail checklist (things to explicitly include)
- Exact **input file triplet** names and where they live (`<expdir>`)
- Exact **output file triplet** names and where they live (`<outdir>/IC_<idate>/`)
- **Temp dirs** recreated each run (`tmpdir`, `ictmpdir`)
- The **optional interpolation** path controlled by `-C` + CDO version caveat
- The **parallelization unit**: vertical levels (137) and that GNU `parallel` is required
- The **boundary**: “Atmosphere ICs only; FESOM ocean ICs are separate”

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% L0 Context (vertical)
  %% =========================

  User["Operator / Scientist"]:::actor
  HPC["HPC Environment (Levante login/compute node)"]:::env

  subgraph "Outside this repo (Coupled model workflow boundary)"
    Upstream["IFS–FESOM short run\n(IC Gen build, >=1 timestep)"]:::externalSys
    Downstream["Branched IFS–FESOM run\n(consumes new IFS ICs)\nNote: FESOM ocean ICs handled separately"]:::externalSys
  end

  subgraph "This repo: HPC batch-style IC generation utility"
    Script["generate_initial_conditions_parallel_only_IFS.sh\n(Orchestrator / Pipeline)"]:::orchestrator

    subgraph "Internal pipeline (ETL over GRIB restarts)"
      P1["1) Parse args / config\n(-t -i -s -e -X -o -C -d)"]:::stage
      P2["2) Workspace setup\n(create/clean tmpdir & ictmpdir)"]:::stage
      P3["3) Split & select GRIB messages\n(grib_filter)"]:::stage
      P4["4) Modify fields / metadata\n(grib_set)"]:::stage
      P5["5) Per-level parallel processing\n(137 upper-air levels)\n(GNU parallel fan-out/fan-in)"]:::stage
      P6{"6) Optional interpolation?\n(flag: -C)"}:::decision
      P6a["6a) Interpolate Gaussian -> spectral\n(cdo gp2sp etc.)\nvars: lnsp, vo, d, t\n(CDO <= 2.4 required for -C)"]:::stageOpt
      P7["7) Assemble / merge final IC products\n(concatenate/merge GRIB msgs)"]:::stage
      Dry{"Dry-run?\n(flag: -d)"}:::decision
      Stop["Print configuration and exit\n(no data processing)"]:::stop
    end

    subgraph "External toolchain (runtime dependencies)"
      ecc["ecCodes CLI\n(grib_filter, grib_set)"]:::tool
      par["GNU parallel"]:::tool
      cdo["CDO (optional)\nrequired only with -C"]:::toolOpt
      mods["Module/Spack toolchain\n(provides ecCodes/parallel/cdo)"]:::env
    end

    subgraph "Data stores / file contracts"
      Expdir[("Input dataset: <expdir>\nIFS restart GRIBs")]:::data
      In1[("ICMGG<expid>+<step>\n(surface grid-point restart)")]:::dataFile
      In2[("ICMSH<expid>+<step>\n(spectral restart)")]:::dataFile
      In3[("ICMUA<expid>+<step>\n(upper-air restart)")]:::dataFile

      Tmp[("<outdir>/tmpdir/\n(ephemeral working dir)")]:::temp
      Ictmp[("<outdir>/ictmpdir/\n(ephemeral working dir)")]:::temp

      Outdir[("Output dataset: <outdir>/IC_<idate>/\nfinal IFS atmosphere ICs")]:::data
      Out1[("ICMSH<expid>INIT\n(spectral core fields)")]:::dataFileOut
      Out2[("ICMGG<expid>INIT\n(surface grid-point fields)")]:::dataFileOut
      Out3[("ICMGG<expid>INIUA\n(upper-air grid-point fields,\nall 137 levels)")]:::dataFileOut
    end

    Docs["README.md\n(overview, usage, deps)"]:::doc
    Runbook["WALKTHROUGH.html\n(step-by-step operations)"]:::doc
  end

  %% =========================
  %% L0 Relationships
  %% =========================
  User -->|"invokesCLI"| Script
  User -->|"reads"| Docs
  User -->|"follows"| Runbook

  User -->|"runsOn"| HPC
  HPC -.->|"moduleLoad/paths"| mods

  Upstream -->|"producesRestartGRIBs"| Expdir
  Expdir -->|"contains"| In1
  Expdir -->|"contains"| In2
  Expdir -->|"contains"| In3

  Script -->|"reads"| Expdir
  Script -->|"writesTemps"| Tmp
  Script -->|"writesTemps"| Ictmp
  Script -->|"writesFinalICs"| Outdir
  Outdir -->|"contains"| Out1
  Outdir -->|"contains"| Out2
  Outdir -->|"contains"| Out3

  Out1 -->|"consumedBy"| Downstream
  Out2 -->|"consumedBy"| Downstream
  Out3 -->|"consumedBy"| Downstream

  %% =========================
  %% Internal pipeline wiring (vertical)
  %% =========================
  Script -->|"starts"| P1
  P1 -->|"check -d"| Dry
  Dry -->|"yes"| Stop
  Dry -->|"no"| P2
  P2 --> P3 --> P4 --> P5 --> P6
  P6 -->|"no (-C not set)"| P7
  P6 -->|"yes (-C set)"| P6a --> P7
  P7 -->|"emit3GRIBs"| Outdir

  %% =========================
  %% Tool invocations (dashed deps)
  %% =========================
  P3 -.->|"invokes"| ecc
  P4 -.->|"invokes"| ecc
  P5 -.->|"invokes"| par
  P6a -.->|"invokes"| cdo

  mods -.->|"provides"| ecc
  mods -.->|"provides"| par
  mods -.->|"provides"| cdo

  %% =========================
  %% File contract focus (dataflow details)
  %% =========================
  In1 -->|"extract/select"| P3
  In2 -->|"extract/select"| P3
  In3 -->|"extract/select"| P3

  P3 -->|"splitMsgs"| Tmp
  P4 -->|"editedKeys"| Ictmp
  P5 -->|"perLevelArtifacts"| Ictmp
  P6a -->|"spectralizedFields"| Ictmp
  Ictmp -->|"merge"| P7

  %% =========================
  %% Click events (from component mapping)
  %% =========================
  click Script "https://github.com/john-amal/ifs_ic_gen/blob/main/generate_initial_conditions_parallel_only_IFS.sh"
  click Docs "https://github.com/john-amal/ifs_ic_gen/blob/main/README.md"
  click Runbook "https://github.com/john-amal/ifs_ic_gen/blob/main/WALKTHROUGH.html"

  %% =========================
  %% Styles (color)
  %% =========================
  classDef actor fill:#ECEFF1,stroke:#607D8B,color:#263238,stroke-width:1px
  classDef env fill:#E0E0E0,stroke:#616161,color:#212121,stroke-width:1px
  classDef externalSys fill:#EDE7F6,stroke:#5E35B1,color:#311B92,stroke-width:1px,stroke-dasharray: 4 3
  classDef orchestrator fill:#BBDEFB,stroke:#1565C0,color:#0D47A1,stroke-width:2px
  classDef stage fill:#E3F2FD,stroke:#1E88E5,color:#0D47A1,stroke-width:1px
  classDef stageOpt fill:#D1C4E9,stroke:#5E35B1,color:#311B92,stroke-width:1px
  classDef decision fill:#FFF3E0,stroke:#EF6C00,color:#E65100,stroke-width:1px
  classDef stop fill:#FFEBEE,stroke:#C62828,color:#B71C1C,stroke-width:1px
  classDef tool fill:#FFE0B2,stroke:#FB8C00,color:#E65100,stroke-width:1px
  classDef toolOpt fill:#F3E5F5,stroke:#8E24AA,color:#4A148C,stroke-width:1px
  classDef data fill:#C8E6C9,stroke:#2E7D32,color:#1B5E20,stroke-width:1px
  classDef dataFile fill:#DCEDC8,stroke:#558B2F,color:#33691E,stroke-width:1px
  classDef dataFileOut fill:#B9F6CA,stroke:#00A152,color:#1B5E20,stroke-width:1px
  classDef temp fill:#D7FFD9,stroke:#2E7D32,color:#1B5E20,stroke-width:1px,stroke-dasharray: 5 3
  classDef doc fill:#FFFDE7,stroke:#F9A825,color:#6D4C41,stroke-width:1px
```
