How ufs-community/uwtools works
Project type and purpose
• Project name: uwtools
• Type: Python library + CLI tool for building and orchestrating scientific workflows (Unified Workflow Tools)
• Purpose: Provide a unified API, configuration management, driver plugins, file-staging, templating and scheduling layers around UFS applications (and general workflow engines like Rocoto).
File structure
– src/uwtools
• cli.py: CLI entrypoint
• api/: high-level Python façade functions (cdeps, chgres_cube, config, driver abstraction, execute, fs, fv3, rocoto, template, etc.)
• config/: parsing, validation and formatting (formats/INI, NML, YAML, SH), using Jinja2 and JSON-schema
• drivers/: plugin modules that wrap invocation of external executables/libraries per component (cdeps, esg_grid, filter_topo, fv3, gsi, mpas, rocoto, etc.)
• fs.py: file-staging utilities (copy, link, makedirs, HPSS support)
• scheduler.py & rocoto.py: workflow graph building & serialization to Rocoto XML
• resources/: JSON-schema definitions & Rocoto schema resources
• logging.py, exceptions.py: cross-cutting support
– docs/: extensive Sphinx documentation & user guide
– notebooks/: interactive demos & tests
– .github/, binder/, recipe/: CI/CD, packaging and deployment scripts
Key insights from README
– CI badges indicate test & release pipelines
a. Main components
1. CLI Layer
– uwtools executable (entrypoint in cli.py)
– ARGS parsing, help output
2. API Layer
– uwtools.api.* modules exposing Python functions
3. Config Management
– Parser, validator, format converters (config/)
– JSON-schema resources under resources/jsonschema
4. Driver Plugins
– uwtools.drivers.* modules wrapping external domain tools
5. File-Staging Service
– uwtools.fs: copy, link, makedirs, HPSS support
6. Execution & Scheduling
– uwtools.execute: serial/parallel execution engine
– uwtools.scheduler + uwtools.resources/rocoto + uwtools.rocoto: build and emit Rocoto XML workflows
7. Support Services
– Logging (uwtools.logging)
– Exceptions (uwtools.exceptions)
8. External Services / Tools
– Domain applications (cdeps, fv3, gsi, etc.) invoked via drivers
– HPSS, local filesystem, MPI, environment modules
b. Relationships and interactions
1. User runs CLI → calls API
2. API loads & validates config → uses config management
3. API calls driver plugins to prepare tasks
4. Scheduler builds task graph → file-staging service schedules data movement
5. Execution engine (serial/parallel) or Rocoto scheduler orchestrates tasks
6. Drivers invoke external binaries; use logging and exceptions for robustness
7. Templates applied during config rendering via Jinja2 sub-component
c. Architectural patterns & design principles
– Modular plugin architecture: clear separation of API vs drivers vs config
– Facade pattern: a unified API over disparate domain tools
– Separation of concerns: CLI vs parsing vs execution vs scheduling
– Schema-driven validation: JSON-schema for config, demarcating extension points
d. Key technologies & frameworks
– Python 3.x, setuptools/pyproject, Click or argparse (in cli.py)
– Jinja2 for templating
– JSON-schema for validation
– Rocoto for workflow scheduling
– HPSS support via htar/hsi in fs module