# ufs-community/uwtools architecture

> Architecture diagram of the GitHub repository ufs-community/uwtools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ufs-community/uwtools
- Repository: https://github.com/ufs-community/uwtools

## Overview

##### Project type and purpose
- Project name: uwtools
- Type: Python library + CLI tool for building and orchestrating scientific workflows (Unified Workflow Tools)
- Purpose: Provide a unified API, configuration management, driver plugins, file-staging, templating and scheduling layers around UFS applications (and general workflow engines like Rocoto).

##### File structure
– src/uwtools
- cli.py: CLI entrypoint
- api/: high-level Python façade functions (cdeps, chgres_cube, config, driver abstraction, execute, fs, fv3, rocoto, template, etc.)
- config/: parsing, validation and formatting (formats/INI, NML, YAML, SH), using Jinja2 and JSON-schema
- drivers/: plugin modules that wrap invocation of external executables/libraries per component (cdeps, esg_grid, filter_topo, fv3, gsi, mpas, rocoto, etc.)
- fs.py: file-staging utilities (copy, link, makedirs, HPSS support)
- scheduler.py & rocoto.py: workflow graph building & serialization to Rocoto XML
- resources/: JSON-schema definitions & Rocoto schema resources
- logging.py, exceptions.py: cross-cutting support
– docs/: extensive Sphinx documentation & user guide
– notebooks/: interactive demos & tests
– .github/, binder/, recipe/: CI/CD, packaging and deployment scripts

##### Key insights from README
– CI badges indicate test & release pipelines

a. Main components
1. CLI Layer
– uwtools executable (entrypoint in cli.py)
– ARGS parsing, help output
2. API Layer
– uwtools.api.* modules exposing Python functions
3. Config Management
– Parser, validator, format converters (config/)
– JSON-schema resources under resources/jsonschema
4. Driver Plugins
– uwtools.drivers.* modules wrapping external domain tools
5. File-Staging Service
– uwtools.fs: copy, link, makedirs, HPSS support
6. Execution & Scheduling
– uwtools.execute: serial/parallel execution engine
– uwtools.scheduler + uwtools.resources/rocoto + uwtools.rocoto: build and emit Rocoto XML workflows
7. Support Services
– Logging (uwtools.logging)
– Exceptions (uwtools.exceptions)
8. External Services / Tools
– Domain applications (cdeps, fv3, gsi, etc.) invoked via drivers
– HPSS, local filesystem, MPI, environment modules

b. Relationships and interactions
1. User runs CLI → calls API
2. API loads & validates config → uses config management
3. API calls driver plugins to prepare tasks
4. Scheduler builds task graph → file-staging service schedules data movement
5. Execution engine (serial/parallel) or Rocoto scheduler orchestrates tasks
6. Drivers invoke external binaries; use logging and exceptions for robustness
7. Templates applied during config rendering via Jinja2 sub-component

c. Architectural patterns & design principles
– Modular plugin architecture: clear separation of API vs drivers vs config
– Facade pattern: a unified API over disparate domain tools
– Separation of concerns: CLI vs parsing vs execution vs scheduling
– Schema-driven validation: JSON-schema for config, demarcating extension points

d. Key technologies & frameworks
– Python 3.x, setuptools/pyproject, Click or argparse (in cli.py)
– Jinja2 for templating
– JSON-schema for validation
– Rocoto for workflow scheduling
– HPSS support via htar/hsi in fs module

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Components
    CLI["CLI Layer"]:::core
    API["API Layer"]:::core
    subgraph "Config Management"
        direction TB
        ConfigParser["Config Parser"]:::core
        FormatConverters["Format Converters"]:::core
        Jinja["Jinja2 Support"]:::core
        Validator["JSON-schema Validator"]:::core
        SchemaResources["Schema Resources"]:::core
    end
    Drivers["Driver Plugins"]:::core

    %% Support Services
    subgraph "Support Services"
        direction TB
        FS["File-Staging Service"]:::support
        Logging["Logging"]:::support
        Exceptions["Exceptions"]:::support
    end

    %% Orchestration
    Execution["Execution Engine"]:::orchestration
    Scheduler["Scheduler"]:::orchestration
    Rocoto["Rocoto Serialization"]:::orchestration

    %% External Systems
    ExternalTools["External Domain Tools"]:::external
    LocalFS["Local FS"]:::external
    HPSS["HPSS"]:::external

    %% Connections
    CLI -->|runs| API
    API -->|loads config| ConfigParser
    API -->|invokes| Drivers
    API -->|uses| FS
    API -->|executes tasks| Execution
    Drivers -->|calls| ExternalTools
    FS -->|manages files| LocalFS
    FS -->|manages files| HPSS
    Execution -->|builds graph| Scheduler
    Execution -->|direct invoke| Drivers
    Scheduler -->|generates XML| Rocoto
    Rocoto -->|orchestrates| ExternalTools

    %% Click Events
    click CLI "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/cli.py"
    click API "https://github.com/ufs-community/uwtools/tree/main/src/uwtools/api/"
    click ConfigParser "https://github.com/ufs-community/uwtools/tree/main/src/uwtools/config/"
    click FormatConverters "https://github.com/ufs-community/uwtools/tree/main/src/uwtools/config/formats/"
    click Jinja "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/config/jinja2.py"
    click Validator "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/config/validator.py"
    click SchemaResources "https://github.com/ufs-community/uwtools/tree/main/src/uwtools/resources/jsonschema/"
    click Drivers "https://github.com/ufs-community/uwtools/tree/main/src/uwtools/drivers/"
    click FS "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/fs.py"
    click Execution "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/api/execute.py"
    click Scheduler "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/scheduler.py"
    click Rocoto "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/rocoto.py"
    click Logging "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/logging.py"
    click Exceptions "https://github.com/ufs-community/uwtools/blob/main/src/uwtools/exceptions.py"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#0000FF,stroke-width:2px
    classDef support fill:#90EE90,stroke:#008000,stroke-width:2px
    classDef orchestration fill:#FFA500,stroke:#FF8C00,stroke-width:2px
    classDef external fill:#D3D3D3,stroke:#696969,stroke-width:2px

    %% Link Styles
    linkStyle 0,1,2,3,4,5,6,7,8,9,10,11 stroke:#222,stroke-width:3px
```
